Visual processing apparatus and visual processing method

ABSTRACT

A visual processing apparatus comprises a spatial processing unit and a visual processing unit. The spatial processing unit is operable to perform a predetermined process to an inputted image signal using surrounding pixels of a target pixel and output an processed signal. The visual processing unit is operable to input the inputted image signal and the processed signal and output an output signal that is visual-processed. The visual processing unit has a property in which, in an area where a value of the image signal is almost equal to a value of the processed signal, a proportion of a change of the output signal to a change of the inputted image signal when the processed signal is fixed to a predetermined level is greater than the proportion when the image signal is equal to the processed signal.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a visual processing apparatus and avisual processing method.

2. Description of the Related Art

Imaging apparatuses, such as a digital still camera for capturing stillimages and a digital video camera for capturing moving images, captureimages in the following manner. In an imaging apparatus, an opticalsystem focuses light and forms an image through exposure control, and asolid-state image sensor, such as a CCD (charge-coupled device) imagesensor or a CMOS (complementary metal oxide semiconductor) image sensor,converts the image to electric signals, which are analogue imagesignals, by photoelectric conversion. In the conventional imagingapparatus, a circuit that performs signal processing, such as analoguefront end processing, then processes the analogue image signals, and anA/D (analogue-to-digital) converter converts the analogue image signalsto digital image data. The digital image data is subsequently subjectedto image processing, such as video gamma correction (gamma correctionwith a gamma of 0.45), knee adjustment, luminance conversion, and colordifference conversion. The processed digital image data is converted todata in a standardized format. More specifically, when the digital imagedata is still image data, the data is converted to, for example, JPEG(Joint Photographic Experts Group) data. When the digital image data ismoving image (video) data, the data is converted to, for example, MPEG(Moving Picture Experts Group) data or DV (Digital Video) data. Thedigital image data in the standardized format is recorded onto arecording medium, such as a memory card, a hard disk, an optical disc,or a magnetic tape.

The whitest point of the data defined by the standardized formatdescribed above (image or video format), which specifically correspondsto the highest luminance level of the data reproduced on a display, isassumed to have a luminance level of 100%. In this case, theconventional imaging apparatus is normally designed to form an imagehaving a dynamic range of luminance levels of 100 to 200%. The dynamicrange of 100% means that signal values (for example, luminance levels)of the image correspond to luminance levels of 0 to 100%. In otherwords, the minimum signal value corresponds to a luminance level of 0%,and the maximum signal value corresponds to a luminance level of 100%.

A first conventional imaging apparatus compresses the dynamic range of acaptured image to eliminate its luminance area exceeding a luminancelevel of 100% through processing called knee adjustment, and obtains animage having a dynamic range of 100% or less.

The dynamic range of an image that is captured by the conventionalimaging apparatus is normally determined by the exposure light amount ofthe optical system (exposure light amount determined by the aperture orshutter speed) and the electric amplification amount of electricsignals, which have been generated by photoelectric conversion.

The conventional imaging apparatus appropriately captures an image(video) of a bright scene (subject) that has a large amount of lighteither by setting a small aperture or by setting a fast shutter speed.This imaging apparatus appropriately captures such an image because theimaging apparatus is provided with a sufficiently large amount of light.Typically, users of imaging apparatuses may intentionally produce“desirable blur” in a captured image (video), or more specificallyintentionally defocus and blur the background of the image (video) bysetting a faster shutter speed and a larger aperture and decreasing thedepth of field. Even when the depth of field is decreased in thatmanner, the imaging apparatus appropriately captures an image (video) inenvironments that have large amounts of light.

Also, users may often set a slower shutter speed and a smaller apertureand increase the depth of field.

In either of the two cases in which the depth of field is increased anddecreased, the imaging apparatus is provided with a sufficiently largeamount of light when capturing an image of a bright scene (a subject ina bright environment), and thus is not required to perform electricamplification of signals. When capturing an image of a bright scene(subject in a bright environment), the conventional imaging apparatusdoes not electrically amplify electric signals, which have beengenerated by photoelectric conversion.

When capturing an image of a dark scene, however, the imaging apparatusis required to maximize the aperture and slow the shutter speed toobtain a sufficiently large amount of light. When the imaging apparatusperforms such a long exposure, the imaging apparatus and the subject maymove during the exposure. Such movement of the imaging apparatus or thesubject may blur images. To prevent blurry images caused by apparatus orsubject movement, the conventional imaging apparatus limits the slowestshutter speed setting to the speed at which such apparatus movement doesnot occur. To compensate for an insufficient amount of exposure light,the conventional imaging apparatus electrically amplifies electricsignals, which have been generated by photoelectric conversion. Suchsignal amplification processing is referred to as “high-sensitivity modeprocessing” or “push processing”. To prevent the S/N (signal-to-noise)ratio from deteriorating and to ensure the quantization resolution ofA/D conversion, the electric push processing is typically performed byamplifying an output of the image sensor of the imaging apparatus usingan analogue circuit. Japanese Unexamined Patent Publication No.2002-135651 describes one technique of push processing, which isperformed by switching the gain of an analogue amplifier circuitaccording to the ISO speed.

When the imaging apparatus obtains image signals using a solid-stateimage sensor, such as a CCD image sensor or a CMOS image sensor, theimage signals have a small dynamic range. In this case, when capturingan image of a scene (subject) that has a large dynamic scene (forexample, when capturing an image of a backlit person outdoors or whencapturing an image of an outdoor landscape from indoors through awindow), the imaging apparatus may fail to appropriately reproduce bothof the dark portion and the bright portion of the subject as clearimages because the luminance levels of the dark portion and the brightportion of the image greatly differ from each other.

To overcome this drawback, a second conventional imaging apparatus usesa solid-state image sensor that can control the charge storage time. Thesecond conventional imaging apparatus superimposes a subject imageformed through a long exposure and a subject image formed through ashort exposure, and consequently obtains a subject image that has alarge dynamic range (see, for example, Japanese Unexamined PatentPublication No. 2005-72965).

A third conventional imaging apparatus uses a solid-state image sensorthat includes high-sensitivity pixels constituting a half of all thepixels of the image sensor and low-sensitivity pixels constituting theremaining half of the pixels. The third conventional imaging apparatussuperimposes an image signal corresponding to a high-sensitivity pixeland an image signal corresponding to a low-sensitivity pixel, which havebeen generated through exposures performed for the same exposure time,and consequently obtains a subject image that has a large dynamic range(see, for example, Japanese Unexamined Patent Publication No.S59-210775).

SUMMARY OF THE INVENTION Technical Problem

However, the conventional imaging apparatus (first conventional imagingapparatus) is only designed to capture an image with a dynamic range ofup to 200%, and compresses the dynamic range of the captured image to100% through knee adjustment. When, for example, the conventionalimaging apparatus is used to capture an image of a face of a backlitperson with a blue sky background, the exposure light amount of theconventional imaging apparatus is set in a manner that the person's facewill have an appropriate luminance level of 70%. In this case, theluminance level of highlights that the conventional imaging apparatuscan handle without saturation (maximum luminance level) would be as lowas about three times the luminance level of the person's face.

However, the dynamic range of a scene (subject) including a clear skycan be easily as high as several hundred percent or more. When theconventional imaging apparatus is used to capture an image of such ascene (subject), the light amount of the image sensor of the imagingapparatus may be saturated. In this case, the imaging apparatus fails toappropriately reproduce a high-luminance portion of the image, such as aclear sky portion.

The second conventional imaging apparatus obtains a subject image thathas an increased dynamic range by using a subject image formed through ashort exposure. However, the short exposure means a small exposure lightamount. With the small exposure light amount, the amount of a signalgenerated by the solid-state image sensor is small. In this case, theS/N ratio of the image would be low. The third conventional imagingapparatus obtains a subject image with an increased dynamic range byusing a signal corresponding to a low-sensitivity pixel. However, theamount of a signal corresponding to a low-sensitivity pixel is small. Inthis case, the S/N ratio of the image would be low in the same manner.

Technical Solution

A visual processing apparatus according to one aspect of the presentinvention comprises a spatial processing unit and a visual processingunit. The spatial processing unit is operable to perform a predeterminedprocess to an inputted image signal using surrounding pixels of a targetpixel and output an processed signal. The visual processing unit isoperable to input the inputted image signal and the processed signal andoutput an output signal that is visual-processed. The visual processingunit has a property in which, in an area where a value of the imagesignal is almost equal to a value of the processed signal, a proportionof a change of the output signal to a change of the inputted imagesignal when the processed signal is fixed to a predetermined level isgreater than the proportion when the image signal is equal to theprocessed signal.

A visual processing method according to another aspect of the presentinvention comprises the following steps.

Performing a predetermined process to an inputted image signal usingsurrounding pixels of a target pixel.

Outputting an processed signal that is obtained by performing thepredetermined process.

Visual-processing using the inputted image signal and the processedsignal as input signals.

Outputting an output signal that obtained by visual-processing.

The visual-processing has a property in which, in an area where a valueof the image signal is almost equal to a value of the processed signal,a proportion of a change of the output signal to a change of theinputted image signal when the processed signal is fixed to apredetermined level is greater than the proportion when the image signalis equal to the processed signal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows main components of an imaging apparatus according to afirst embodiment of the present invention;

FIG. 2 shows main components of a signal processing unit 31;

FIG. 3 shows examples of compression characteristics used in dynamicrange compression, which are selected by a dynamic-range compressionunit 2001;

FIG. 4A shows a signal processing unit 90 included in a conventionalimaging apparatus, and FIG. 4B shows a conversion characteristic used indynamic range compression;

FIG. 5 is a diagram describing a predetermined image scene 201 and thedynamic range of signals corresponding to different portions of thescene;

FIG. 6 is a diagram describing a predetermined image scene 201 and thedynamic range of signals corresponding to different portions of thescene;

FIG. 7 is a diagram describing a predetermined image scene 201 and thedynamic range of signals corresponding to different portions of thescene;

FIG. 8 shows main components of a signal processing unit 31′ accordingto a first modification of the first embodiment;

FIG. 9 shows examples of dynamic-range compression characteristics;

FIG. 10 is a diagram describing a predetermined image scene 201 and thedynamic range of signals corresponding to different portions of thescene;

FIG. 11 shows main components of a signal processing unit 31″ accordingto a second modification of the first embodiment;

FIG. 12 is a diagram describing a predetermined image scene 201 and thedynamic range of signals corresponding to different portions of thescene;

FIG. 13 shows main components of a signal processing unit 31′″ accordingto a third modification of the first embodiment;

FIG. 14 is a diagram describing a predetermined image scene 201 and thedynamic range of signals corresponding to different portions of thescene;

FIG. 15 is a diagram describing a predetermined image scene 201 and thedynamic range of signals corresponding to different portions of thescene;

FIG. 16 is a flowchart showing an imaging method according to the firstembodiment of the present invention;

FIG. 17 shows main components of the signal processing unit 31A;

FIG. 18 shows main components of the dynamic-range compression unit2001:

FIG. 19 shows examples of dynamic-range compression characteristics;

FIGS. 20A and 20B show photographic images from which the advantageouseffects of the imaging apparatus are obvious;

FIG. 21 shows examples of characteristics of data stored in atwo-dimensional LUT;

FIG. 22 shows examples of dynamic-range compression characteristics usedin a dynamic-range compression process using avisualization-characteristic-based technique;

FIG. 23 shows examples of dynamic-range compression characteristics usedin a dynamic-range compression process using avisualization-characteristic-based technique;

FIG. 24 shows main components of an imaging apparatus using atwo-dimensional LUT;

FIG. 25 shows tone conversion characteristics with profiles used when aninput range of a two-dimensional LUT is 400% (4.0);

FIG. 26 shows tone conversion characteristics with profiles used whenthe input range of the two-dimensional LUT is 200% (2.0);

FIG. 27 shows main components of an imaging apparatus that includes arange adjustment unit 2601;

FIG. 28 shows main components of an imaging apparatus that includes arange adjustment unit 2701;

FIG. 29 shows main components of an imaging apparatus that includes apeak adjustment unit 2801;

FIG. 30 shows main components of the peak adjustment unit 2801, thedynamic-range compression unit 2001, and a representative valuedetection unit 3163 included in the imaging apparatus;

FIG. 31 shows main components of an imaging apparatus according to afourth embodiment of the present invention;

FIGS. 32A and 32B are timing charts chronologically describing theoperation of the imaging apparatus;

FIG. 33 shows main components of a dynamic-range increasing unitincluded in the imaging apparatus;

FIG. 34 is a diagram describing the dynamic-range increasing unitincluded in the imaging apparatus;

FIG. 35 is a diagram describing a subject whose image is captured by theimaging apparatus;

FIG. 36 is a diagram describing a dynamic-range compression unitincluded in the imaging apparatus;

FIG. 37 is a flowchart showing the operation of the imaging apparatus;

FIG. 38 shows another structure of the dynamic-range compression unitincluded in the imaging apparatus;

FIG. 39 is a diagram describing the operation of the dynamic-rangecompression unit included in the imaging apparatus;

FIGS. 40A and 40B show photographic images from which the advantageouseffects of the dynamic-range compression unit included in the imagingapparatus are obvious;

FIG. 41 shows main components of a dynamic-range increasing unitincluded in an imaging apparatus according to a fifth embodiment of thepresent invention;

FIGS. 42A and 42B are diagrams describing apparatus movement correction;

FIG. 43 is a diagram describing an apparatus movement detection unitincluded in the imaging apparatus;

FIG. 44 shows main components of a dynamic-range increasing unitincluded in an imaging apparatus according to a sixth embodiment of thepresent invention;

FIGS. 45A and 45B are diagrams describing apparatus movement correctionand subject movement correction;

FIG. 46 shows main components of a dynamic-range increasing unitincluded in an imaging apparatus according to a seventh embodiment ofthe present invention; and

FIGS. 47A and 47B are diagrams describing apparatus movement correctionand subject movement correction.

BEST MODE FOR CARRYING OUT THE INVENTION

An imaging apparatus according to embodiments of the present inventionwill now be described with reference to the drawings.

First Embodiment

1.1 Structure of the Imaging Apparatus

FIG. 1 shows the structure of an imaging apparatus 100.

The imaging apparatus 100 includes an optical system 1, an analoguesignal processing unit 2, a digital signal processing unit 3, and anexposure meter 7.

The optical system 1 includes an imaging lens 11, an aperture 12, and animaging unit 13. The imaging lens 11 focuses light from a subject P1.The aperture 12 adjusts the amount of reflection light from the subjectP1, which has been focused through the imaging lens 11. The imaging unit13 outputs an image signal A according to the adjusted light amount(light amount adjusted by the aperture 12) and the light accumulationtime.

The analogue signal processing unit 2 includes a correlated doublesampling (CDS) circuit 21 and an analogue-to-digital (A/D) converter 23.

The A/D converter 23 converts analogue image (video) signals to 12-bitdigital image (video) signals (signals C) corresponding to pixels havinglevels of tone ranging from 0 to 4095, and outputs the digital image(video) signals. For ease of explanation, the imaging apparatus 100includes a gain control amplifier (GCA) 22 as shown in FIG. 1. However,the GCA 22 is not an essential component of the imaging apparatus 100.The GCA 22 may include the CDS circuit.

The digital signal processing unit 3 includes a signal processing unit31. The processing units of the digital signal processing unit 3 includea flash illumination control unit, a codec (coder/decoder), a card I/Fand a display control unit, and a control unit 34. The control unit 34controls the operations of all or some of the other processing units ofthe digital signal processing unit 3.

The functional blocks of the imaging apparatus 100 may be connected toone another by a bus as shown in FIG. 1.

1.1.1 Structure of the Signal Processing Unit 31

FIG. 2 shows the structure of the signal processing unit 31. The signalprocessing unit 31 includes a dynamic-range increasing processing unit2000 and a dynamic-range compression unit 2001. The dynamic-rangeincreasing processing unit 2000 includes a dynamic-range increasing unit2002 and a register 2004.

The dynamic-range increasing unit 2002 is specifically formed by adigital multiplier. The dynamic-range increasing unit 2002 multipliesoutput signals (signals C) of the analogue signal processing unit 2 witha predetermined dynamic range increase ratio (for example, 2 or 4), andstores the multiplication result into the register 2004. The dynamicrange increase ratio is set at a smaller value based on a signal Mag1 asthe value of the aperture of the optical system is set smaller (or asthe received light amount, which is measured with the exposure meter, isset smaller by controlling the shutter speed etc.). When, for example,the value of the aperture 12 is set at ½, the dynamic range increaseratio is set at 2. When, for example, the value of the aperture 12 isset at ¼, the dynamic range increase ratio is set at 4.

The register 2004 is a storage unit for storing the multiplicationresult of the output resolution (12 bits) of the A/D converter 23 andthe maximum value (for example, 8) of the dynamic range increase ratiowhile maintaining the resolution of the multiplication result. Theoutput (signal C) of the A/D converter 23 is assumed to have 12 bitsrepresenting a dynamic range of 150%. In this case, when the maximumvalue of the dynamic range increase ratio is 8, the signals, whosedynamic range has been increased, have 15 bits representing a dynamicrange of 1200%. In this case, the register 2004 may be formed by a15-bit register that can store the theoretical maximum value of themultiplication. Alternatively, the register 2004 may be formed by amemory that can store a plurality of signals (for example, signalscorresponding to one image (one frame)) while assuming the resolution ofthe 15-bit register as one unit.

The dynamic-range compression unit 2001 converts the multiplicationresult (signals D) stored in the register 2004 using a predeterminedconversion characteristic to signals E, and outputs the signals E. Theconversion characteristic is determined based on a signal Mag2, whichcorresponds to the set aperture value of the optical system.

Examples of the compression characteristic used in dynamic-rangecompression, which are selectively used by the dynamic-range compressionunit 2001, will now be described with reference to FIG. 3.

The horizontal axis in FIG. 3 indicates an input dynamic range (%),which is the dynamic range of input signals D. The maximum dynamic rangeof an input signal that can be stored in the register 2004 is 1200%. Thevertical axis in FIG. 3 indicates an output dynamic range (%), which isthe dynamic range of output signals E.

According to conversion characteristics L0 to Lmax, the input and theoutput are the same when the input value is smaller than or equal to avalue corresponding to point P (80%). According to these conversioncharacteristics, the value of each output signal E changes according tothe value of its input signal when the input value is greater than 80%.

The dynamic-range compression unit 2001 selects the conversioncharacteristic L0 based on a selection signal Mag2 when the value of theaperture 12 is set at 1 as described above. The dynamic-rangecompression unit 2001 selects the conversion characteristic L1, withwhich the dynamic range of up to four times can be output, when thevalue of the aperture 12 is set at ¼. The dynamic-range compression unit2001 selects the conversion characteristic Lmax, with which the dynamicrange of up to eight times can be output, when the value of the aperture12 is set at ⅛. In the first embodiment, as shown in FIG. 3, the dynamicrange of a signal may be increased within a range from the input maximumdynamic range (150%) corresponding to the conversion characteristic L0to the input maximum dynamic range (1200%) corresponding to theconversion characteristic Lmax.

1.2 Operation of the Imaging Apparatus

The operation of the imaging apparatus 100 will now be described.

In the imaging apparatus 100, the control unit 34 sets the value of theaperture 12 at a predetermined value (for example, ⅛). The aperturevalue is preset based on an output value of the exposure meter 7 or bythe user setting.

The control unit 34 sets the dynamic range increase ratio of thedynamic-range increasing unit 2002 at 8 (the inverse of the aperturevalue ⅛) based on a control signal Mag1. Based on a control signal Mag2,the control unit 34 selects the conversion characteristic Lmax, which isused by the dynamic-range compression unit 2001.

When the imaging apparatus 100 starts operating, the imaging unit 13obtains signals A, whose value depends on the value (aperture value) ofthe aperture 12. The analogue signal processing unit 2 then converts thesignals A through A/D conversion to signals C, and outputs the signalsC. The signals C are input into the signal processing unit 31 shown inFIG. 2, and are multiplied by a predetermined dynamic range increaseratio of, for example, 8. As a result, signals D are generated andoutput. The output signals D have a value less than or equal to a valuecorresponding to the maximum dynamic range of 1200%. The output signalsD can be stored in the register 2004 without overflow.

With the conversion characteristic Lmax, the signals D are eventuallyconverted to output signals E. The output signals E, whose value dependson the value of the output signals D, are then output.

The imaging apparatus 100 and the signal processing unit 31 in the firstembodiment perform the processing described above.

1.3 Advantageous Effects of the Imaging Apparatus

The advantageous effects of the imaging apparatus 100 will now bedescribed in comparison with a conventional imaging apparatus.

Conventional Imaging Apparatus

FIG. 4A shows a signal processing unit 90 included in the conventionalimaging apparatus. FIG. 4B shows a conversion characteristic used indynamic-range compression performed by the conventional imagingapparatus. The conventional signal processing unit 90 shown in FIGS. 4Aand 4B differs from the signal processing unit 31 included in theimaging apparatus 100 in the following points. The signal processingunit 90 does not include the dynamic-range increasing unit 2002 and theregister 2004. The signal processing unit 90 has only the singleconversion characteristic. The signal processing unit 90 has an inputvalue range of as small as 150%.

First Conventional Technique

FIG. 5 shows signal values that change when the conventional imagingapparatus shown in FIG. 4B compresses the dynamic range of signalscorresponding to a predetermined scene using the dynamic-rangecompression characteristic of the conventional imaging apparatus.

The left part of FIG. 5 shows the structure of the predetermined scene201.

A point 204 is included in a cloud and sky portion of the scene 201. Apoint 202 is a predetermined point included in a face portion of thescene 201. A portion 203 is a main subject portion of the scene 201.Hereafter, the point 204 included in the cloud and sky portion isassumed to have a value of a brightest portion of the scene 201.

The exposure condition of the imaging apparatus is set in a manner thatoutput signals corresponding to the main subject portion 203 will have aluminance level of 70% with respect to a possible dynamic range (150%)of an output signal. This exposure condition is referred to as the“exposure condition 1”.

Under the exposure condition 1, signals A (signals at point A in FIG. 1)(the same applies hereafter) corresponding to the point 204, whichactually have a luminance level of 500%, are saturated. With any signalprocessing, signals C corresponding to the sky 204 (the point 204) of animage resulting from digital conversion of the signals A will besaturated. This is because signal values within a range RA shown in theright part of FIG. 5 would never be recovered in subsequent processing.Among signals A, signals with a dynamic range of 150% and less (withinranges RB and RC) are converted through A/D conversion to signals Cwhile maintaining their luminance level and without being saturated. Thesignals C are then subjected to dynamic range compression using theconversion characteristic shown in FIG. 4B. The signals C that arewithin the range RC are output as signals E with an appropriateluminance level of 70%.

However, the conventional technique fails to achieve an appropriateluminance level of the face portion 202 (for example, a luminance levelof 70%) (object 1) and at the same time fails to prevent values of thepoint 204 included in the sky and cloud portion and its vicinity portionfrom being saturated (object 2).

Second Conventional Technique

The second conventional technique limits the exposure light amount toprevent signals A within the range RA (range of light amounts of 150% ormore) in FIG. 3 from being saturated. For example, the conventionalimaging apparatus may use the exposure condition 1 and additionally setthe exposure light amount of the optical system to ¼. This exposurecondition is referred to as the “exposure condition 2”.

In this case, signals A corresponding to the point 204 in FIG. 5 have avalue of 125%. Thus, even the signals A corresponding to the brightestportion are not saturated. However, under the exposure condition 2, theconventional technique fails to obtain output signals E with anappropriate luminance level (for example, a luminance level of 70%)corresponding to the face point 202 (object 1) although the conventionaltechnique successfully prevents signals corresponding to the sky andcloud portion 204 (point 204) from being saturated (object 2).

Third Conventional Technique

The exposure condition 2 used with the second conventional technique issimilar to an exposure condition used for a dark scene with aninsufficient exposure light amount. Push processing, which is applied toimage signals obtained under an exposure condition for a dark scene withan insufficient light amount, may be applied to the exposure condition2.

The signal values that change when image signals are processed by theelectrical amplification used in the push processing (amplificationperformed by the analogue signal processing unit (GCA)) (referred to asmethod 3) will now be described with reference to FIG. 6.

When the image signals (signals A) are subjected to the exposure withthe aperture value of ¼ (under the exposure condition 2), the sky andcloud portion 204, which actually has a luminance level of 500%, willhave a luminance level of 125%. In this case, signals corresponding tothe sky and cloud portion 204 are not saturated. The bar graph in FIG. 6(signals B and C) (signals B correspond to signals at point B in FIG. 1and signals C correspond to signals at point C in FIG. 1) (the sameapplies hereafter) indicates that the 4× push processing of signals(signals B) performed by the CDS circuit 22 recovers the luminance levelof the face portion 202 to an appropriate luminance level of 70%.However, signals corresponding to the point 204, which actually have aluminance level of 500%, are saturated when the input maximum value ofthe A/D converter 23 is 150%. As a result, signals within the range RA(150% or more) (for example, the saturated sky portion) in FIG. 5 wouldnever be recovered at the time when the image signals are input into theA/D converter 23. As a result, the third conventional technique alsofails to achieve an appropriate luminance of the face point 202 (forexample, a luminance level of 70%) (object 1) and at the same time failsto prevent signals corresponding to the sky and cloud portion point 204or its vicinity portion from being saturated (object 2).

As described above, with any of the first to third conventionaltechniques described above, the conventional imaging apparatus thatincludes the signal processing unit 90 shown in FIG. 4 fails to achieveboth the object 2 of obtaining unsaturated outputs corresponding to thesky point 204 and its vicinity portion and the object 1 of achieving anappropriate luminance level (a luminance level of 70%) of the mainsubject portion within the range RC.

Advantageous Effects of the Imaging Apparatus 100

The advantages effects of the imaging apparatus 100 will now bedescribed with reference to FIG. 7. The imaging apparatus 100 is assumedto capture an image of the same scene 201 as described above.

In the same manner as the second and third conventional techniques, theimaging apparatus 100 first sets the exposure condition 2 (with theaperture value of ¼) for the aperture 12 to prevent signals Acorresponding to the bright portion (sky and cloud portion including thepoint 204) from being saturated. Under the exposure condition 2, signalscorresponding to the point 204 included in the sky and cloud portion,which is the brightest portion, are subjected to the exposure with theaperture value of ¼. In this case, signals corresponding to the point204, which can actually have a value of 500%, will have a value of 125%.The signals corresponding to the point 204 are not saturated.

The signals A with a value of 125% are converted to digital signals(signals C) while maintaining their signal values of 125% without beingamplified by the analogue processing unit. The signals C (125%) aresubjected to 4× (the inverse of ¼) multiplication performed by thedynamic-range increasing unit 202. Through the multiplication, thesignals C with a value of 125% are converted to signals D with a valueof 500%.

The signals D with this signal value (500%), which is within theincreased dynamic range, are converted to signals E2 using theconversion characteristic L1 shown in FIG. 3 (the value of each signalE2 is calculated as 80+20*(500−80)/(600−80)). In this manner, theimaging apparatus 100 prevents the signals corresponding to the point204 of the brightest portion (or its vicinity portion) from beingsaturated (object 2).

The signals A corresponding to the face point 202, which can actuallyhave a dynamic range of 70%, have a value of 17.5%. The signals A withthis value (17.5%) are converted to digital signals (signals C) throughA/D conversion while maintaining their values. The signals C (17.5%) arethen subjected to 4× push processing, which is performed by thedynamic-range increasing unit 2002. Signals D into which the signals Care converted recover the signal value of 70% (=17.5*4). The signals Dhave the value of 70%, which is smaller than a value corresponding to aninflection point P (80%) of the conversion characteristic L1. As aresult, signals E into the which the signals D are converted by thedynamic-range compression unit 2001 have a value of 70%. The signals Ewith the value of 70% are then output.

The imaging apparatus 100 outputs signals corresponding to the mainsubject with an appropriate luminance level (luminance level of 70%). Inother words, the imaging apparatus 100 achieves the object 2.

As described above, the imaging apparatus 100 of the first embodimentpresets a smaller aperture value of the aperture 12 to prevent thereceived light amount from being saturated, and then subjects digitalsignals (signals C) resulting from A/D conversion to various processing.

In particular, the imaging apparatus 100 sets the dynamic range increaseratio at a value (1, 2, or 4) that is inversely proportional to thevalue (1, ½, or ¼) of the aperture 12. This enables signal values to berecovered to values that can be achieved under the exposure condition 1.Additionally, the imaging apparatus 100 selects the dynamic-rangecompression conversion characteristic according to the aperture value(1, ½, or ¼). Thus, the imaging apparatus 100 converts even outputsignals D that are within the increased dynamic range (150 to 1200%) tooutput signals E having changing tone levels.

As described above, the imaging apparatus 100 of the first embodimentobtains image data that prevents even a bright portion, such as a pointincluded in a sky and cloud portion of an image, from being saturated.The imaging apparatus 100 of the first embodiment further obtains anappropriate luminance level of a point corresponding to a face position.In other words, the imaging apparatus 100 achieves both the objects 1and 2.

First Modification

The first embodiment describes the case in which the dynamic-rangecompression conversion characteristics L0 to Lmax are selected accordingto the preset aperture amount (exposure light amount). Alternatively,the dynamic-range compression unit 2001 may select the dynamic-rangecompression characteristic Ln according to a maximum value of signals Dcorresponding to the entire image (first modification).

FIG. 8 shows the structure of a signal processing unit 31′ according tothe first modification. In FIG. 8, the processing units that are thesame as the units described in the first embodiment are given the samereference numerals as those components. The signal processing unit 31′according to the first modification differs from the signal processingunit 31 shown in FIG. 2 only in its parameter determination unit 800.

The parameter determination unit 800 includes a peak detection unit 802.

The peak detection unit 802 receives signals D corresponding to oneimage that are stored in the register 2004 of the dynamic-rangeincreasing unit 2000, and extracts a maximum value (or peak value) (in avalue of the brightest portion) of the image data corresponding to oneimage.

The peak detection unit 802 then outputs a parameter MLn to thedynamic-range compression unit 2001. The parameter MLn is used to setthe conversion characteristic corresponding to the extracted maximumvalue. When, for example, the maximum value of the predetermined entireimage is 500%, the peak detection unit 802 outputs the selectionparameter ML2.

The dynamic-range compression unit 2001 selects the correspondingconversion characteristic Ln (L2 in the figure) shown in FIG. 9according to the selection parameter MLn, which is output from theparameter determination unit 800.

The advantageous effects of the first modification will now be describedusing the same scene as described above shown in FIG. 10.

Signals D corresponding to the sky and cloud portion 204, which is thebrightest portion, have the same output values as described in the firstembodiment before or at the timing at which the value of 500% is output.

The parameter determination unit 800 extracts the maximum value of 500%of image data corresponding to one image (image data corresponding toone frame) that is formed using signals D, and outputs a parameter ML2.Based on the parameter ML2, the dynamic-range compression unit 2001selects the conversion characteristic L2. The dynamic-range compressionunit 2001 converts the signals D stored in the register 2004 using theconversion characteristic L2, and outputs the signals.

The signals D with a value of 500% corresponding to the brightestportion are converted using the conversion characteristic L2. Thesignals with the maximum value of 500% of the image are converted tosignals E with a value of 100%. The signals E with a value of 100% arethen output. With the structure described above, the signals D (500%)with the maximum value corresponding to the sky and cloud portion areconverted to signals with a value smaller than 100%(80+20*(500−80)/(600−80)). The structure according to the firstmodification shown in FIG. 8 achieves both the objects 1 and 2 describedabove, and further obtains outputs whose peak value of the entire imageis 100%.

The conversion timings may be adjusted to use the conversioncharacteristic extracted for image data of one image by the parameterdetermination unit 800 to convert subsequent image data of the sameimage. For example, a delay circuit with a delay of an imagecorresponding to one screen may be arranged to precede the dynamic-rangecompression unit 2001.

Second Modification

Alternatively, the aperture amount of the aperture 12 and the dynamicrange increase ratio may be set according to the peak value of theentire image that is detected by the parameter determination unit 800 ofthe first modification (second modification).

The structure according to the first modification uses the aperturevalue of the aperture 12 preset by a separate means (for example, ¼). Inthat case, signals A corresponding to the brightest portion of the scene201 stored in the imaging unit 13 (output from the imaging unit 13) donot have a value as high as the maximum dynamic range of 150% (signalsonly have a value of up to 125%). In this case, digital signals Cobtained for the scene at the imaging timing do not have values as highas the maximum input dynamic range (quantization resolution) of the A/Dconversion. Thus, the structure according to the first modificationfails to effectively use the maximum input dynamic range (quantizationresolution).

The structure according to the second modification sets the aperturevalue larger to increase the exposure light amount in a manner that thepeak value of signals D will be as high as the maximum dynamic range ofthe input. This structure effectively uses the resolution of the A/Dconverter 23.

FIG. 11 shows the structure of a signal processing unit 31″ according tothe second modification.

A peak detection unit 802′ outputs a signal (ML2) for selecting theconversion characteristic L2 according to the peak value of signals D inthe same manner as the structure of the first modification.

As shown in FIG. 11, the peak detection unit 802′ of the secondmodification outputs a setting signal X1 for controlling the apertureamount of the optical system 1 and a setting signal X2 for setting thedynamic range increase ratio of the dynamic-range increasing processingunit.

Based on the setting signal X1, the aperture 12 is controlled in amanner that the peak value of 500% of signals D will be 150%, which isthe maximum input value of the A/D conversion (the value of the signalX1 is 150/500 (=0.3)).

As shown in FIG. 1, the control unit 34 may receive the setting signalX1, and control the aperture 12 based on the setting signal X1.

Based on the setting signal X2, the dynamic range increase ratio is setat 500/150, which is the inverse of the aperture amount. Morespecifically, the signals X1 and X2 have values determined using amountscontradictory to each other. The value of the setting signal X2 maydirectly be the peak value or may directly be the dynamic range increaseratio.

The advantageous effects of the second modification will now bedescribed with reference to FIG. 12.

The structure according to the second modification does not change thepeak value of the image (500%), but uses a larger value as the apertureamount of the aperture 12 (150/500). Therefore, signals A have values ashigh as the input maximum value of 100% of the A/D conversion. Morespecifically, the imaging apparatus 100 according to the secondmodification improves the relationship between the aperture value andthe dynamic range increase ratio as compared with the imaging apparatusof the first modification, which presets the aperture value of theaperture 12 at ¼. As a result, the imaging apparatus 100 of the secondmodification increases the peak value of analogue signals to the inputmaximum range of the A/D conversion. Therefore, the imaging apparatus100 of the second modification effectively uses the maximum quantizationresolution of the A/D conversion.

Third Modification

Unlike the second modification, the parameter determination unit 800 maynot use the maximum value of the entire image as a reference. Instead,the parameter determination unit 800 may use a representative value of apredetermined portion (main subject portion) as a reference to determinethe dynamic range increase ratio or the aperture value of the aperture(including the adjustment value of the exposure light amount asdescribed above). The imaging apparatus 100 of the third modificationmay further adjust the aperture amount of the aperture 12 or the dynamicrange increase ratio in a manner that a representative value F of theimage signals corresponding to the predetermined portion will coincidewith a predetermined reference value.

The aperture value of the aperture 12 may be preset (at 3/10) and thedynamic range increase ratio may be preset (at 10/3) in a manner thatthe maximum output value of signals D corresponding to the predeterminedportion is 70% at timing tk shown in the first embodiment and the firstand second modifications. However, depending on the circumstances suchas the amount of incident light in the vicinity of the subject, thestructures of the first embodiment and the first and secondmodifications may fail to have the output value of the predeterminedportion (for example, the subject portion) close to the target value of70% at timings subsequent to timing tk. The structure according to thethird modification enables the output value of the predetermined portionto be adjusted toward the target value even in such a case.

The structure of the third modification will now be described withreference to FIG. 13.

As shown in FIG. 13, a parameter determination unit 800′″ includes arepresentative value detection unit 1201. The representative valuedetection unit 1201 determines a representative value (F1) of apredetermined portion of image data corresponding to one image (imagedata corresponding to one frame). The predetermined portion is a presetcentral portion of the image. The predetermined portion may contain asubject designated by the user. In this example, the maximum value ofsignals D corresponding to pixel positions included in the predeterminedportion is determined as the representative value F1.

The representative value detection unit 1201 outputs (A) a settingsignal X1 for setting the aperture amount of the optical system or (B) asetting signal X2 for setting the dynamic range increase ratio that isused by the dynamic-range increasing unit 2002 in a manner that therepresentative value F1 determined at the predetermined timing ismaintained close to its target value at timings subsequent to thepredetermined timing.

The peak detection unit 802 outputs a parameter MLx, which is used bythe dynamic-range compression unit 2001 to select the conversioncharacteristic Lx based on the peak value (P) of the entire image.

Advantageous Effects of the Third Modification

The operation of the imaging apparatus 100 according to the thirdmodification and its advantageous effects will now be described usingthe same scene 201 as described above with reference to FIG. 14.

The representative value detection unit 1201 is assumed to extract arepresentative value of 70% for a predetermined portion at apredetermined timing in the manner described above. The dynamic rangeincrease ratio is set at 10/3 at timing tk. The aperture value is set at3/10.

The representative value detection unit 1201 is then assumed to extracta representative value of the central portion (portion 203) of imagedata obtained at predetermined timing tn. More specifically, therepresentative value detection unit 1201 is assumed to extract arepresentative value of 50% for the central portion of the image data.

The representative value detection unit 1201 outputs either a settingsignal X1 or a setting signal X2 in a manner that the representativevalue of 50% will be output as a target appropriate value (target valueof 70%) for the same portion of the image data (signals D) obtained attiming tn+1.

When the signal X1 is output:

(A) In response to the setting signal for setting the aperture amount ofthe optical system, the representative value detection unit 1201 sets asignal X1 for newly setting the aperture amount at a value that is 70/50times the aperture amount set at timing tn.

When the signal X2 is output:

(B) In response to the setting signal for setting the dynamic rangeincrease ratio used by the dynamic-range increasing unit 2002, therepresentative value detection unit 1201 outputs a signal X2 for settingthe dynamic range increase ratio at a value that is 70/50 times thedynamic range increase ratio set at timing t1 to the dynamic-rangeincreasing unit 2002.

In the example shown in FIG. 14, the representative value detection unit1201 outputs the setting signal X2 for newly setting the dynamic rangeincrease ratio at 10/3*70/50.

In this case, the maximum value of the signals A and C corresponding tothe central portion of the image data obtained at timing tn+1 (theoutput of the representative value detection unit 1201) is maintained tobe 15%, whereas the signals D corresponding to the central portion ofthe image data have a value of 70% based on 15(%)*(10/3*70/50)=70(%),after conversion is performed using the dynamic range increase ratioupdated based on data obtained at timing tn. The entire image data isconverted using the predetermined conversion characteristic Ln2 based onthe peak value of the signals D of the entire image obtained at timingtn.

This structure maintains the representative value of the predeterminedportion to be a value close to the target value of 70%. When, forexample, processing continuous image data (moving image etc.), theimaging apparatus with this structure prevents the luminance level of apredetermined portion of image data from changing drastically andenables the luminance level of the predetermined portion to changegradually.

Although not shown, the dynamic range is compressed with conversioncharacteristics determined according to the maximum value (P) of signalsD of the entire image. This prevents the output of signals (signals E)corresponding to the brightest portion from being saturated.

According to the third modification, when the signal X2 for updating thedynamic range increase ratio is generated, or specifically in case (B)described above, the output signals (signals E) are obtained in a mannerthat the representative value (luminance level) of the predeterminedportion (central portion) will be the target output value of 70% withoutcausing mechanical delays of the aperture 12.

According to the third modification, when the signal for changing theaperture amount of the optical system is generated, or specifically incase (A) described above, the signals A to E change in the manner shownin FIG. 15.

In this case, the values of analogue signals A corresponding to thesubject vicinity portion (portion 203 in the above example), which areaccumulated in the imaging unit 13, are smaller in the scene newlyobtained at each of the timings to and tn+1. However, the imagingapparatus of the third modification sets an appropriate exposure amountof 21% at timing tn+1. In total, the imaging apparatus of the thirdmodification sets the appropriate exposure light amount and the dynamicrange increase ratio centering on the subject.

Although the present embodiment describes the case in which the maximumvalue of the signals D corresponding to the predetermined portion isused as the representative value, the representative value may be avalue other than the maximum value of the signals D. For example, therepresentative value may be an average value of the signals D, a medianvalue of the signals D, or a value obtained by eliminating certainsignals out of the signals D.

1.4 Imaging Method

The imaging apparatus 100 and the signal processing units 31, 31′, 31″,or 31′″ of the first embodiment (including the first to thirdmodifications) (the same applies hereafter) may use an imaging methodincluding processing that is performed by various processing units.

The imaging method according to the first embodiment of the presentinvention (imaging method used in the imaging apparatus 100) will now bedescribed with reference to a flowchart shown in FIG. 16.

According to the imaging method of the first embodiment, the exposurecondition of the imaging unit (image sensor) 13 is set in a manner thata highlight portion of a scene is not saturated based on, for example,values corresponding to a brightest portion of an immediately precedingimage of the scene captured with the live view function or the like(S100).

Under the set exposure condition, image signals are generated by theimaging unit (image sensor) 13 (S101).

The image signals are converted to digital image data by the A/Dconverter 23 (S102).

The digital image data is subjected to the linear dynamic range increaseand the push processing performed simultaneously by the dynamic-rangeincreasing unit 2002 in a manner that the subject main portion will havea predetermined luminance level (for example, a luminance level of 70%)(S103).

The image data is then subjected to the dynamic range compression basedon the conversion characteristic determined according to the luminancelevel of the brightest portion of the scene (S104).

Through the processing from steps S100 to S104 of the imaging method ofthe present embodiment, image data of the main subject portion willrecover its predetermined luminance level and the highlight portion willretain its tone levels without saturation even with the preset smallerexposure light amount.

The imaging apparatus and the imaging method according to the presentembodiment produce the advantageous effects unique to the presentinvention by performing electric signal amplification for a bright sceneas well as performing dynamic range increase, although the electricsignal amplification is conventionally considered unnecessary for abright scene with a sufficiently large amount of light. Morespecifically, the imaging apparatus and the imaging method of thepresent embodiment intentionally set a smaller exposure light amount fora bright scene by adjusting the aperture amount and the shutter speed,and perform the electric signal amplification in combination with thedynamic range increase.

According to the present invention, the exposure condition for a brightscene is set to be the condition conventionally considereddisadvantageous for a bright scene. Image signals generated by theimaging unit 13 under the set exposure condition are converted todigital image signals by A/D conversion. The digital image signal arethen subjected to the dynamic range increase, which is performed throughdigital processing. Consequently, the imaging apparatus and the imagingmethod of the present invention obtain image signals that achieve apredetermined luminance level of a main subject portion of the scene andat the same time prevent a highlight portion of the scene from beingsaturated.

In the present embodiment, the optical system 1 may have any structureas long as the optical system 1 can control the exposure light amount ofthe imaging unit 13. Although the present embodiment describes the casein which the imaging lens 11 is formed by a single lens, the imaginglens 11 may be formed by a plurality of lenses.

The shutter, which is not shown, may be a mechanical shutter, or may bean electronic shutter that adjusts the amount of light by adjusting thedriving timing of the imaging unit (image sensor) 13.

The A/D converter 23 may be arranged separately from the analogue signalprocessing unit 2, or may be incorporated in the digital signalprocessing unit 3.

The imaging unit (image sensor) 13 may be an image sensor with anotherstructure, such as a CMOS image sensor. The structure of the imagingunit (image sensor) 13 should not be limited to a single-sensorstructure, but may be a triple-sensor structure.

The dynamic-range increasing unit 2002 should not be limited to amultiplier. It is only required that the dynamic-range increasing unit2002 can linearly increase the dynamic range of a signal through digitalprocessing. For example, the dynamic-range increasing unit 2002 may beformed by a lookup table (LUT) that has an output value with a three-bithigher resolution than an input value when, for example, the maximumdynamic range increase ratio is 8. When, for example, the dynamic rangeincrease ratio is the power of two, a shifter (bit shifter) may be used.

Although the present embodiment describes the case in which thedynamic-range compression unit 2001 compresses the dynamic range ofsignals with values corresponding to luminance levels exceeding 80% anddoes not compress the dynamic range of signals with values correspondingto luminance levels of 80% and less, the present invention should not belimited to such a structure. It is only required that the compressioncharacteristic of the dynamic-range compression unit 2001 be such thatthe dynamic range of signals is converted with any compressioncharacteristic that changes according to a maximum value of an inputvalue.

For example, the dynamic-range compression unit 2001 may convert thedynamic range of signals corresponding to an input value of 20% or lessto signals D with small differences between their tone levels. It ispreferable that the conversion characteristic is set to maintain orcoordinate differences between tone levels of signals corresponding to amain subject vicinity portion.

Although the second modification describes the case in which the dynamicrange increase is performed in a manner that the luminance level of thebrightest portion of the scene coincides with the maximum dynamic rangeof the imaging unit (image sensor) 13 (the maximum value of image signalvalues output from the imaging unit 13), the present invention shouldnot be limited to such a structure. The luminance level of the brightestportion of the scene may be lower than the maximum dynamic range of theimaging unit 13. Alternatively, the relationship between the luminancelevel of the brightest portion of the scene and the dynamic range of theimaging unit 13 may be set to permit signals corresponding to thebrightest portion to be saturated to a degree at which the image is notsubstantially seen degraded.

The compression characteristic of the dynamic-range compression unit2001 may not necessarily be set based on an output of the peak detectionunit 802. To calculate an approximate value, the maximum dynamic rangeof the imaging unit (image sensor) 13 may be multiplied by the exposurelight amount correction value described above. The calculatedapproximate value may then be set in the dynamic-range compression unit2001.

Second Embodiment

An imaging apparatus (camera) according to a second embodiment of thepresent invention will now be described.

2.1 Structure of the Imaging Apparatus

The imaging apparatus of the present embodiment differs from the imagingapparatus 100 of the first embodiment in its signal processing unit 31Ashown in FIG. 17, which replaces the signal processing unit 31. Theother structure of the imaging apparatus of the present embodiment isthe same as the structure of the imaging apparatus 100 of the firstembodiment. The components of the imaging apparatus of the presentembodiment that are the same as the components of the imaging apparatus100 of the first embodiment are given the same reference numerals asthose components, and will not be described.

FIG. 17 shows the structure of the signal processing unit 31A of thepresent embodiment.

As shown in FIG. 17, the signal processing unit 31A differs from thesignal processing unit 31 of the first embodiment in its face detectionunit (characteristic image portion detection unit) 1701 and a luminancedetection unit 1702, which are additional components not included in thesignal processing unit 31 of the first embodiment. The functional blocksof the signal processing unit 31A that are the same as the functionalblocks of the signal processing unit 31 (functional blocks with the samereference numerals as in FIG. 2) will not be described.

The face detection unit 1701 analyzes characteristic amounts of an imagethat is formed using image data (image signals), and detects a person'sface portion of the image. The face detection unit 1701 then outputsinformation about the coordinates representing the face portion andabout the range of the face portion to the luminance detection unit1702.

The luminance detection unit 1702 detects the luminance level of theface portion based on the information about the coordinates representingthe face portion and about the range of the face portion, which isoutput from the face detection unit 1701.

The dynamic-range increasing unit 2002 receives an output of theluminance detection unit 1702 and image signals output from a firstsignal processing unit 311A, and increases the dynamic range of theimage signals in a manner that the face portion, which is detected bythe luminance detection unit 1702, will have an appropriate luminancelevel.

2.2 Operation of the Imaging Apparatus

The operation of the imaging apparatus of the present embodiment willnow be described with reference to FIG. 12 described above.

The imaging apparatus is assumed to capture an image of a scene whoseportions have the same dynamic range as the dynamic range of theportions of the scene described in the first embodiment. Thus, FIG. 8 isalso referred to in describing the operation of the imaging apparatus.

The imaging apparatus is assumed to capture an image of the same sceneas described above. In the first embodiment, the imaging apparatus 100captures an image of the scene by assuming that the main subject portion202, such as the face portion, is included in the main subject vicinityportion 203, which is about the center of the scene 201. In this case,it may not be possible for the imaging apparatus of the first embodimentto always use the optimal condition to capture the image. The imagingapparatus of the present embodiment eliminates the above assumption, andalways uses the optimal condition to capture an image of the scene. Morespecifically, the imaging apparatus of the present embodiment performsexposure control based only on values corresponding to a brightestportion of the scene 201.

The imaging apparatus of the present embodiment will hereafter bedescribed in detail.

The control unit 34 calculates an exposure light amount AP1 in a mannerthat a highlight portion of the subject scene is not saturated in theimaging unit (image sensor) 13 based on, for example, measurement valuesof an exposure meter (not shown) or values corresponding to a brightestportion of an immediately preceding image of the scene (valuescorresponding to a brightest portion included in the sky and cloudportion 204) captured by the imaging unit (image sensor) 13 with thelive view function or the like. The exposure light amount is actuallycontrolled (by adjusting the aperture amount and the shutter speed) tobe the exposure light amount AP1. In this case, the sky and cloudportion 204, which is the brightest portion, has the luminance level of150%, and is not saturated in the imaging unit (image sensor) 13.Therefore, the signal values of image signals generated by the imagingunit (image sensor) 13 are not saturated. When the exposure light amountof the imaging apparatus of the present embodiment is set to theexposure light amount AP1, the person's face portion 202 will have aluminance level of 21% in the same manner as in the first embodiment.

The face detection unit 318 analyzes an image that is formed using theimage data (image signals) resulting from A/D conversion, and detectsthe person's face portion of the image and obtains the coordinatesrepresenting the face portion of the image. The face detection unit 318then outputs information about the coordinates representing the faceportion of the image to the luminance detection unit 319.

The luminance detection unit 319 calculates the average luminance levelof the face portion of the image, which is formed using the image data(image signals), based on the information about the coordinatesrepresenting the face vicinity portion of the image output from the facedetection unit 318. The face detection unit 318 calculates the averageluminance level of skin portions of the face vicinity portion excludinghair, eyes, etc. (the average luminance level is referred to as the“face luminance level”, which is specifically 21% in the presentexample). The luminance detection unit 319 then calculates the dynamicrange increase ratio (70(%)/21(%)=3.33), with which the face luminancelevel will be an appropriate luminance level of 70%. The luminancedetection unit 319 sets the calculated dynamic range increase ratio inthe dynamic-range increasing unit 2002.

The dynamic-range increasing unit 2002 increases the dynamic range ofthe image signals based on the dynamic range increase ratio, which isset by the luminance detection unit 319. More specifically, thedynamic-range increasing unit 2002 increases the dynamic range of theimage signals in a manner that the main subject portion 202 will have aluminance level of 70% based on 21(%)*10/3=70(%), and the sky and cloudportion 204 will have a luminance level of 500% based on150(%)*10/3=500(%). The dynamic-range increasing unit 2002 then outputsthe image signals, whose dynamic range has been increased, to theregister 2004.

The dynamic-range increasing unit 2002 and the register 2004 for storingimage signals whose dynamic range has been increased by thedynamic-range increasing unit 2002 can handle image signals with valuescorresponding to luminance levels up to 1200% without saturating thesignals. Consequently, the imaging apparatus of the present embodimentachieves an appropriate luminance level of the main subject portion 202of the image and at the same time prevents the sky and cloud portion 204of the image from being saturated.

The operation of the dynamic-range compression unit 2001 is the same asthe operation described in the first embodiment, and will not bedescribed.

As described above, the imaging apparatus (camera) of the presentembodiment obtains image signals that achieve a predetermined luminancelevel of a face portion of a scene with higher precision than in thefirst embodiment and retain tone levels of a highlight portion of thescene without saturation.

The face detection unit 318 may detect a face portion using variousother algorithms known in the art. For example, the face detection unit318 may use pattern matching based on learning to detect the faceportion using the face line or the arrangement of the mouth, eyes, andnose, or may use color information, such as skin color information, todetect the face portion.

Third Embodiment

A dynamic-range compression unit 2001 included in an imaging apparatus(camera) according to a third embodiment of the present invention willnow be described. In the first and second embodiments, the dynamic-rangecompression unit 2001 performs dynamic range compression using the tonecurves with the nonlinear characteristics shown as in FIG. 3 (conversioncharacteristics having inflection points larger than and smaller thanthe input range value of 80%). With this structure, the imagingapparatus effectively uses the increased large dynamic range (forexample, the dynamic range of 1000%), which is increased by thedynamic-range increasing unit 2002, and thereby captures an image of ascene in a manner that a main subject portion of the scene, such as aface portion, has a predetermined luminance level and a highlightportion of the scene with a large dynamic range retains its tone levelswithout saturation.

However, the dynamic-range compression unit 2001 shown in FIG. 2 doesnot compress the dynamic range of signals with values corresponding toluminance levels of 80% and less, and compresses the dynamic range ofsignals with values corresponding to luminance levels exceeding 80% andup to 1200% to the dynamic range of 80 to 100%. The dynamic rangecompression performed with this dynamic range compression characteristicretains tone levels of highlight portions with a wide range of luminancelevels. However, because the input and output characteristic curve ofthe dynamic range compression (straight line in FIG. 3) has an extremelysmall gradient, the contrast of the highlight portion decreases greatlyafter the dynamic range compression is performed with the above dynamicrange compression. In FIG. 3, the input and output characteristic curveof the dynamic range compression of the highlight portion has a gradientof 0.0176, which is calculated as (100−80)/(1200−80)=0.0176. Thegradient of 0.0176 indicates a high compression ratio.

In other words, the gradient of the input and output characteristiccurve of the dynamic range compression of the highlight portion issmaller as the dynamic-range compression unit 2001 compresses thedynamic range more. As a result, although the highlight portion retainsits changing tone levels after the tone level conversion, the tonevalues (signal values) of the highlight portion differ only too slightlyfrom one another. In this case, the image captured by the imagingapparatus of the present embodiment differs insignificantly from asaturated image that would be captured with a conventional technique. Inother words, the image captured by the imaging apparatus of the presentembodiment can be seen as a saturated image. This can be the fundamentalproblem of the dynamic range compression of the imaging apparatus.Despite this problem, the imaging apparatus of the present invention isstill obviously advantageous over the saturation occurring with theconventional technique (the phenomenon in which a highlight portion of acaptured image is saturated), because the gradient of the input andoutput characteristic curve of the dynamic range compression of ahighlight portion of a scene would not be extremely small when thedynamic range compression performed with the dynamic range compressionconversion characteristic shown in FIG. 3 is applied to the sceneportions with luminance levels of several hundred percent or less.

To solve this fundamental problem, the tone characteristic of eachposition on an image may be changed according to the luminance level ofa vicinity portion of spatial signals determined for each position onthe image. One such technique is described in International PublicationNo. WO 2005/027041. More specifically, the patent document describes atechnique for using a vicinity portion of processing target pixels of animage. With this technique, for example, the histogram of a vicinityportion of processing target pixels of an image is measured, and thetone curve used for the target pixels is determined based on thedistribution of values of the histogram. Alternatively, the averageluminance level of the vicinity portion of the processing target pixelsis calculated, and the tone curve used for the target pixels isdetermined according to the luminance level of the vicinity portion. Theconventional technique is ineffective when input signals have alreadybeen saturated. Therefore, the conventional technique alone will notproduce the advantageous effects of the present invention.

When the technique described above is applied to the dynamic-rangecompression unit 2001, the imaging apparatus (camera) of the presentinvention will solve the problem of low contrast in the highlightportion by minimizing the contrast decrease caused by the dynamic rangecompression. More specifically, the imaging apparatus of the presentinvention converts large-dynamic-range information, which is informationabout an image with the increased large dynamic generated by thedynamic-range increasing unit 2002, to small dynamic range information,which is information about an image with a dynamic range of 100% orless, without visually causing almost no loss of the image information.Consequently, the imaging apparatus of the present embodiment maximizesthe effects of the dynamic range increase of the present invention.

The imaging apparatus according to the present embodiment will now bedescribed.

3.1 Structure of the Imaging Apparatus

The imaging apparatus of the present embodiment includes thedynamic-range compression unit 2001 having the structure shown in FIG.18 instead of the structure of the dynamic-range compression 2001included in the imaging apparatus of the first and second embodiments.The other structure of the imaging apparatus of the present embodimentis the same as the structure of the imaging apparatuses of the first andsecond embodiments. The components of the imaging apparatus of thepresent embodiment that are the same as the components of the imagingapparatuses of the first and second embodiments are given the samereference numerals as those components, and will not be described.

As shown in FIG. 18, the dynamic-range compression unit 2001 includes avicinity luminance detection unit 3161 and a dynamic tone correctionunit 3162.

The vicinity luminance detection unit 3161 receives image signals outputfrom the register 2004. The vicinity luminance detection unit 3161detects a representative value (for example, an average value) ofluminance levels (pixel values) of a vicinity portion of processingtarget pixels included in an image formed using the image signals. Thevicinity luminance detection unit 3161 then outputs the detectedrepresentative value to the dynamic tone correction unit 3162.

The dynamic tone correction unit 3162 receives the image signals outputfrom the register 2004 and the representative value output from thevicinity luminance detection unit 3161. The dynamic tone correction unit3162 subjects the image signals to dynamic tone correction using a tonelevel conversion characteristic. The tone level conversioncharacteristic is represented by a dynamic range compression curve thatchanges according to an output of the vicinity luminance detection unit3161. More specifically, the dynamic tone correction unit 3162 performsthe dynamic tone correction with the tone characteristic that changesaccording to positions on the image (spatial positions (or the luminancelevel of a vicinity portion determined for each vicinity portion)). Thedynamic tone correction unit 3162 outputs the image signals, whosedynamic range has been compressed, to a second signal processing unit317B.

The first signal processing unit 311A receives image signals output fromthe analogue signal processing unit 2, and subjects the image signalsoutput from the analogue signal processing unit 2 to signal processesincluding white balance correction, pixel interpolation, colorcorrection, noise reduction, and enhancement, and outputs the processedsignals to the dynamic-range increasing unit 2002. Alternatively, thesecond signal processing unit 317B may perform the signal processesincluding white balance correction, pixel interpolation, colorcorrection, nose reduction, and enhancement. It is only required thateach of the signal processes including white balance correction, pixelinterpolation, color correction, noise reduction, and enhancement beperformed either by the first signal processing unit 311A or by thesecond signal processing unit 317B. The signal processes may beperformed in any order, and any of the signal processes may be assignedto the first signal processing unit 311A or the second signal processingunit 317B.

3.2 Operation of the Imaging Apparatus

The operation of the imaging apparatus of the present embodiment willnow be described. The operation parts of the imaging apparatus otherthan the operation of the dynamic-range compression unit 2001 are thesame as described in the above embodiments, and will not be described.

The operation of the dynamic-range compression unit 2001 of the presentembodiment will be described with reference to FIG. 19.

The dynamic-range compression unit 2001 has five tone curves (tone levelconversion characteristic curves) shown in FIG. 19. The dynamic-rangecompression unit 2001 selects one of the five tone curves according tothe luminance level of a vicinity portion of processing target pixels.

As shown in FIG. 19, the dynamic-range compression unit 2001 selects thecurve a when the vicinity portion of the target pixels is the darkest,the curve c when the vicinity portion of the target pixels is thebrightest, and the curve b when the vicinity portion of the targetpixels has an intermediate luminance level between the darkest and thebrightest cases. Although the present embodiment describes the case inwhich the dynamic-range compression unit 2001 has the five tone levelconversion characteristic curves for ease of explanation, the number ofthe characteristic curves should not be limited to five. It ispreferable that the dynamic-range compression unit 2001 actually uses inthe tone level conversion of image signals as many tone level conversioncharacteristic curves as curves that can be assumed substantiallycontinuous to one another. For example, the dynamic-range compressionunit 2001 may have several tens of tone level conversion characteristiccurves.

The dynamic-range compression unit 2001 uses the tone level conversioncharacteristic curve a when processing image signals corresponding to adark vicinity portion of target pixels included in an image (forexample, the face vicinity portion 203 in FIG. 5). Thus, the tone levelsof image signals corresponding to the person's face portion areconverted to tone levels (signal values) corresponding to an appropriateluminance level of about 70%. The dynamic-range compression unit 2001uses the tone level conversion characteristic curve c when processingimage signals corresponding to an extremely bright vicinity portion oftarget pixels included in the image (for example, the sky and cloudportion 204 in FIG. 5). Thus, the image signals corresponding to the skyand cloud portion 204 are converted to image signals with a sufficientlylarge number of tone levels and a sufficiently high contrast (gradient).In this manner, through the dynamic range compression performed by thedynamic-range compression unit 2001, the imaging apparatus of thepresent embodiment (1) sets the large gradient of the input and outputcharacteristic indicating bright range values for a bright portion, and(2) sets the large gradient of the input and output characteristicconversion for a dark portion. Consequently, the imaging apparatus ofthe present embodiment obtains an image with a high definition (imagesignals) that is seen (extremely) natural by human eyes.

The dynamic range compression technique described above is based on thevisualization characteristics of humans, and this technique is called“visual processing”. Such a visualization-characteristic-based technique(visual processing) is based on the visualization characteristics of theeyes of humans. The human eyes (1) increase sensitivity when viewing abright portion and (2) decrease sensitivity when viewing a dark portion.More specifically, the dynamic range compression technique considers thevisualization characteristics of humans that the relative contrastbetween a predetermined position and its vicinity position is importantin viewing the predetermined position irrespective of the actualluminance levels of output signals corresponding to the predeterminedposition. As a result, the imaging apparatus of the present embodimentthat performs the visualization-characteristic-based processingeffectively compresses the large dynamic range of an image and preventsan image from being seen unnatural.

The imaging apparatus of the third embodiment compresses an increaseddynamic range of 1000% or more of an increased-dynamic-range image, orfor example, an increased dynamic range of several thousands percent ofan increased-dynamic-range image, which is obtained by the dynamic-rangeincreasing unit 312, to a dynamic range of 100% or less without visuallydegrading the increased large dynamic range of the image.

3.3 Advantageous Effects of the Imaging Apparatus

The advantageous effects of the imaging apparatus of the presentembodiment will now be described using images that are actually capturedby the imaging apparatus (camera) of the present embodiment and aconventional imaging apparatus (camera).

FIG. 20A shows an image captured through conventional camera processingwhose exposure is controlled in a manner that a person's face, which isa main subject, has an appropriate luminance level. FIG. 20B shows animage captured by the imaging apparatus that incorporates thedynamic-range compression unit 316 of the present embodiment in thesignal processing unit 31 of the second embodiment (an image capturedthrough camera processing of the present embodiment).

The images shown in FIGS. 20A and 20B have almost the same luminancelevel in dark portions, such as the shaded portion of the waterwheel,and also have almost the same luminance level in the face portion.However, the images have different luminance levels in bright portions,such as the sky and cloud portion and the person's clothing portion inwhich sunlight is reflected. In detail, the bright portions of the imageshown in FIG. 20A are saturated, and overexposed and fail toappropriately reproduce color. In contrast, the bright portions of theimage shown in FIG. 20B are seen natural as the bright portions of theimage retain their continuously changing tone levels as well as thecontours and the contrast of the sky and the cloud.

The images in FIGS. 20A and 20B also differ from each other in the colorof the sky. The sky portion of the image in FIG. 20A is overexposed andfails to appropriately reproduce color, whereas the sky portion of theimage in FIG. 20B appropriately reproduces color of the blue sky. Thecomparison between the two images reveals that the dynamic-rangecompression unit 2001 of the present embodiment effectively compressesthe increased dynamic range of image signals to the small dynamic rangeof 100% and obtains a natural image while effectively using the largedynamic range increased by the dynamic-range increasing unit 2002.

Although the present embodiment describes the case in which thedynamic-range compression unit 2001 has the structure shown in FIG. 18,the present invention should not be limited to this structure of thedynamic-range compression unit 2001. The dynamic-range compression unit2001 may have another structure. It is only required that thedynamic-range compression unit 2001 can change the input and outputconversion characteristic according to spatial positions on an image (orthe luminance level of a vicinity portion of predetermined targetpixels). For example, the structure of the dynamic-range compressionunit 316 of the present invention may be modified to use othertechniques known in the art, such as a Retinex-theory-based techniqueand local histogram equalization, with which the dynamic-rangecompression unit 2001 can have substantially the same advantageouseffects as described above.

The dynamic tone correction unit 3162 may not calculate the toneconversion characteristic curve or the output value according to thevicinity luminance value, but may store the tone conversioncharacteristic curve or the output value in a two-dimensional LUT.

FIG. 21 shows examples of characteristics of data stored in thetwo-dimensional LUT.

FIG. 21 shows examples of conversion characteristic data of dynamicrange compression performed using the two-dimensional LUT when the inputdynamic range of the dynamic-range compression unit 2001 is increased to800% (8.0 in FIG. 21) (the same applies hereafter).

In the figure, a broken line indicated by arrow k represents theconversion characteristic of luminance levels of an image used in thedynamic range compression performed using the two-dimensional LUT. Thisconversion characteristic corresponds to the knee function of thecamera. The conversion characteristic (conversion characteristicindicated by the broken line k) represents values output when an inputequal to the vicinity luminance value is input into the dynamic tonecorrection unit 3162.

In the figure, each curve drawn with a solid line indicates theconversion characteristic that uses the visualization characteristicsdescribed above (characteristics that change according to the luminancelevel of the vicinity portion). In the same manner as described above,the curve a, which indicates the conversion characteristic based on thevisual characteristics, is selected when the vicinity portion ofprocessing target pixels is the darkest, and the curve c, whichindicates the conversion characteristic based on the visualcharacteristics, is selected when the vicinity portion of the processingtarget pixels is the brightest.

When the vicinity luminance value is smaller (darker), a curve locatedmore leftward is selected accordingly. This means that signalscorresponding to a darker portion of a natural image are converted in amanner that signals with small input values will retain contrast. Asdescribed above, human eyes do not perceive absolute luminance levelsbased on predetermined pixel values but perceive luminance levels ofpixels based on the relative contrast within the closed range of a darkportion.

This particular feature will now be described in detail with referenceto FIG. 22.

In a portion (sky and cloud portion) whose input value is 80% (0.8 ormore), a vicinity portion (local area) of a point included in the skyportion has a high luminance level that is the same as the luminancelevel of a point included in the sky portion. Thus, the input valuecorresponding to this portion (sky and cloud portion) and the luminancelevel of its vicinity portion are substantially equal to each other. Forinputs that are substantially equal to their vicinity luminance values(inputs corresponding to vicinities A, B, and C in the figure), thecurves (curves a to c) of the conversion characteristics are set in amanner that the contrast ratio will be maintained at each of the pointson the broken line K (vicinities A, B, and C in the figure) (in a mannerthat the gradient of each of the conversion curves a, b, and c will begreater than the gradient of the curve K (kinked line in FIG. 22), whichis defined by the knee function of the camera in the vicinities A, B,and C).

As shown in FIG. 22, the gradient of the conversion curve in thevicinity (indicated by a circle in the figure) of the point ofintersection between the curve K and each of the conversion curves a toc, which is defined by the knee function of the camera, is set the sameas the gradient of the straight line that elongates on the point oforigin. The gradient of the conversion curve is set in this manner tocompress the dynamic range while visually maintaining the contrast.Alternatively, the conversion curve (tone conversion curve) may be setto have a sharp gradient at positions corresponding to a bright portion.In this case, the contrast in the bright portion increases further asshown in FIG. 23.

This means the conversion characteristic changes a value of inputdigital signal more dynamically in the case a value of the digitalsignal is close to a value of the vicinity luminance value. If inputvalue (Vi) is close to a vicinity luminance value (VLVi), an amount ofchange (delta Vi) for value (Vi) is more than an amount of change (deltaVj) for the input value (Vj) which is far from the VLVi (i.e. in thecase value Vj is around 0).

The conversion characteristic data stored in the LUT may be obtainedfrom an external source based on a selection signal MLn (aperturevalue). In this case, the imaging apparatus has the structure shown inFIG. 24.

In this case, profiles stored in the LUT define the conversioncharacteristic curves with the characteristics shown in FIG. 25, whichare used when the input range is 400% (4.0), and the conversioncharacteristic curves with the characteristics shown in FIG. 26, whichare used when the input range is 200% (2.0).

Alternatively, the two-dimensional LUT 3162 may not store a plurality ofprofiles each of which corresponds to a different dynamic range increaseratio. The two-dimensional LUT 3162 may store only a single profile. Inthis case, the imaging apparatus is required to include a rangeadjustment unit 2601 as shown in FIG. 27. More specifically, the rangeadjustment unit 2601 adjusts the dynamic range when the input signalvalue (signal value of the signal D) is 80% or more, and may input thesignal value (signal value of the signal D), whose dynamic range hasbeen adjusted, into the two-dimensional LUT 3162.

For example, the two-dimensional LUT 3162 may have a profile with anincrease ratio of 1 (the conversion characteristic profile that sets theratio of the output range to the input range as 1). The output range ofa signal output from the register 2004 is assumed to be 4.0, and theinput range of the two-dimensional LUT is assumed to be 1.0.

In this case, the range adjustment unit 2601 may have an input value Dof the signal D, which is generated by increasing the dynamic range ofthe input value greater than 0.8 (signal value of the signal D). In thiscase, the range adjustment unit 2601 may have an output value Dout,which is calculated as Dout=0.8+0.2*(D−0.8)/(4.0−0.8).

As shown in FIG. 28, the two-dimensional LUT may have a profile with anincrease ratio of 8 (the conversion characteristic profile that sets theratio of the output range to the input range as 8). The output range ofa signal output from the register 2004 is assumed to be 4.0, and theinput range of the two-dimensional LUT is assumed to be 8.0. In thiscase, the range adjustment unit 2701 may have an input value D of thesignal D, which is generated by increasing the dynamic range of theinput value four times. In this case, the range adjustment unit 2701 mayhave an output value Dout, which is calculated asDout=0.8+7.2*(D−0.8)/(4.0−0.8), where D>0.8.

Alternatively, the imaging apparatus may include a range adjustment unit2701, which adjusts the dynamic range based on a signal Mag2 as shown inFIG. 28.

Alternatively, the imaging apparatus may include a peak adjustment unit2801 as shown in FIG. 29. The peak adjustment unit 2801 detects the peakvalue of image data corresponding to one image (image data correspondingto one frame) that is formed using the signals D whose dynamic range hasbeen increased, and adjusts the input value greater than 80% (0.8) basedon the maximum value (Pmax) according to the detected peak value. Morespecifically, the peak adjustment unit 2801 may adjust the signal valuein a manner that the maximum value X of the signal will be increased tothe input range of the two-dimensional LUT 3162 after the dynamic rangeof the signal is increased by the dynamic-range increasing unit 2002.For example, when the two-dimensional LUT 3162 stores a profile with adynamic range increase ratio of 8 (the conversion characteristic profilethat sets the ratio of the output range to the input range as 8), thepeak adjustment unit 2801 may have an output value Dout, which iscalculated as Dout=0.8+7.2*(D−0.8)/(Pmax−0.8), where D is the inputvalue of the peak adjustment unit 2801.

Alternatively, the peak adjustment unit 2081 shown in FIG. 29 may adjustthe peak value based on a signal Mag2.

Although the maximum value of the input value of the two-dimensional LUT3162 is 8.0 (800%) in the above example, the input value of thetwo-dimensional LUT 3162 may be increased to 16.0 (to 1600%) or 32.0 (to3200%) as long as the input has the same characteristic as describedabove (the gradient of the input and output conversion coefficient(input and output conversion curve) has the tendency of the curvedefined by the knee function of the camera when the input value greaterthan 0.8 is equal to its vicinity luminance value).

The data (output data) stored into the two-dimensional LUT 3162 may notdirectly be an output value with respect to an input value and avicinity luminance value. The data may be a coefficient from which theoutput value is calculated.

The size of the vicinity portion of an image that is processed by thevicinity luminance detection unit 3161 may be set according to arequired level of the visual effect. It is preferable that the vicinityluminance detection unit 3161 uses a vicinity portion (portionconsisting of pixels that are in the vicinity of processing targetpixels) with a certain size to achieve a required level of the visualeffect. For example, when the image is an XGA (eXtended Graphics Array)image (1024 by 768 pixels), it is preferable to set the vicinity portion(portion consisting of pixels that are in the vicinity of processingtarget pixels) having at least an area corresponding to 80 by 80 pixels.

The vicinity luminance detection unit 3161 may use a low-pass filter toobtain information about a vicinity portion of processing target pixels,such as an FIR (finite impulse response) low-pass filter or an IIR(infinite impulse response) low-pass filter, which is normally used togenerate unsharp signals. Alternatively, the imaging apparatus mayinclude a representative value detection unit 3163 as shown in FIG. 30.The representative value detection unit 3163 is used to adjust the localcontrast of the image data.

Fourth Embodiment

An imaging apparatus according to a fourth embodiment of the presentinvention will now be described.

The imaging apparatus according to the fourth embodiment obtains imagesignals corresponding to a plurality of sub-frames by subjecting theimage sensor to a plurality of sequential divisional exposures (shortexposures), which are performed to prevent an output of the A/Dconverter from being saturated. Through the exposures, the image sensorreads an optical image of a subject, which has been formed using lightfocused by an optical system having the light amount adjustmentfunction, and obtains the image signals corresponding to the pluralityof sub-frames. The imaging apparatus of the present embodiment thenconverts the sub-frame image signals corresponding to the plurality ofsub-frames by A/D conversion, which is performed by the A/D converter,to generate digital sub-frame images. The imaging apparatus thenincreases the dynamic range of each pixel of the digital sub-frameimages and adds up the same pixels of the different images. As a result,the imaging apparatus forms an increased-dynamic-range image, which isan image with an increased dynamic range.

The imaging apparatus with this structure obtains the plurality ofsub-frame images by subjecting the image sensor to the plurality ofsequential divisional exposures. In this case, the sub-frame imagesobtained by the imaging unit are dark. Thus, outputs resulting from A/Dconversion corresponding to even a bright portion of the subject areless likely to be saturated. The imaging apparatus increases the dynamicrange of each pixel of the plurality of sub-frame images, which havebeen obtained through the plurality of divisional exposures, and adds upthe same pixels of the plurality of sub-frame images. As a result, theimaging apparatus forms an increased-dynamic-range image. Therefore, theimaging apparatus prevents an output image from being saturated andenables the output image to have an appropriately high luminance level.Consequently, the imaging apparatus of the present embodiment obtains alarge dynamic range image while preventing the S/N ratio of the imagefrom deteriorating. The imaging apparatus of the present embodimentappropriately captures an image of even a scene including a backlitperson outdoors without saturating a bright portion of the image, suchas a sky or cloud portion, even when a dark portion of the image, suchas a person's face portion, is corrected to have an appropriateluminance level.

4.1 Structure of the Imaging Apparatus

The structure of the imaging apparatus 400 according to the fourthembodiment will now be described with reference to FIGS. 31 to 33. FIG.31 is a block diagram showing the structure of the imaging apparatus 400of the present embodiment. FIGS. 32A and 32B are timing chartschronologically describing the operation of the imaging apparatus 400 ofthe present embodiment. FIG. 33 is a block diagram showing the structureof the dynamic-range increasing unit 411 included in the imagingapparatus 400. The number of divisional exposures into which theexposure for capturing one scene image is divided is determinedaccording to a required dynamic range increasing ratio. For ease ofexplanation, the exposure is assumed to be divided into four divisionalexposures in the present embodiment.

As shown in FIG. 31, the imaging apparatus 400 of the present embodimentincludes an optical system 416, an image sensor (imaging unit) 404, andan A/D converter 405. The optical system 416 includes an optical system402 and an aperture 403. The optical system 402 includes a lens andother components. The aperture 403 is arranged at the back of(subsequent to) the optical system 402, and adjusts the amount ofincident light. The image sensor (imaging unit) 404 is, for example, aCCD image sensor or a CMOS image sensor. The image sensor (imaging unit)404 converts an optical image of a subject P2, which has been formedwith the light focused by the optical system 416, to electric signals.The A/D converter 405 converts analogue images, which are output fromthe image sensor (imaging unit) 404, to digital images. The imagingapparatus 400 further includes an operation unit 407, a system controlunit 408, an optical system 416, and a drive unit 406. The operationunit 407 receives an operation command input by a user. The systemcontrol unit 408 controls the entire imaging apparatus according tosettings output from the operation unit 407. The optical system 416operates based on a command signal provided from the system control unit408. The drive unit 406 drives the image sensor 404 and the A/Dconverter 405. The imaging apparatus 400 further includes an imageprocessing unit 410, a display unit 414, and a storage unit 415. Theimage processing unit 410 generates a dynamic-range-increased image(image signals), which is an image with an increased dynamic range,based on RGB images (image signals), which are images resulting from A/Dconversion performed by the A/D converter 405. The display unit 414displays an image (image formed using the image signals), which has beenprocessed by the image processing unit 410. The storage unit 415 storesthe image processed by the image processing unit 410.

The optical system 416 includes the optical system 402 and the aperture403, and forms the optical image of the subject P2 onto the image sensor(imaging unit) 404.

The optical system 402 includes a lens and other components, which areknown in the art.

According to an exposure condition set by an exposure condition settingunit 409, which will be described later, the aperture 403 adjusts theamount of incident light before the image sensor 404 is subjected to theplurality of divisional exposures. More specifically, the aperture 403adjusts the amount of incident light according to a predeterminedexposure time in a manner that a subject main portion (for example, aperson's face portion) will have a predetermined luminance level. Inthis manner, the optical system 402 and the aperture 403 correspond tothe light amount adjusting function of the optical system 416.

The image sensor 404 includes an array of pixels each having the samesensitivity. The image sensor 404 receives incident light for thepredetermined exposure time. The amount of the incident light isadjusted by the aperture 403. The image sensor 404 outputs analogueimage signals, which are proportional to the amount of exposure lightthat is the incident light amount multiplied by the predeterminedexposure time.

The A/D converter 405 converts analogue image signals output from theimage sensor 404 to digital images (digital image signals).

The drive unit 406 drives the optical system 402, the aperture 403, theimage sensor 404, and the A/D converter 405 based on a command signalprovided from the system control unit 408.

The system control unit 408 includes a central processing unit (CPU), aread only memory (ROM), and a random access memory (RAM). The CPUexecutes predetermined processing according to programs. The ROM is aread-only storage device that stores programs and data. The RAM is arewritable memory for temporarily storing data.

The operation unit 407 outputs the settings performed by the user, suchas an imaging mode set by the user, to the system control unit 408,which will be described later. The user may selectively set the imagingmode according to, for example, a subject scene. The imaging apparatusmay have, for example, an indoor imaging mode, an outdoor imaging mode,a sport scene imaging mode, a portrait scene imaging mode, and alandscape scene imaging mode. In particular, the user selects theoutdoor imaging mode for an outdoor scene. In this case, the incidentlight has a large dynamic range. Thus, the exposure condition is set ina manner that a main subject portion (person's face portion) of thescene that may be backlit and dark will have an appropriate luminancelevel. Even in this case, the imaging apparatus is required to processimage signals in a manner that an output corresponding to a brightsubject portion of the scene (for example, a sky and cloud portion) isnot saturated.

The system control unit 408 includes the exposure condition setting unit409. The system control unit 408 reads RGB image signals, which areoutput from the A/D converter 405. The exposure condition setting unit409 then calculates an appropriate exposure condition. Morespecifically, the system control unit 408 sets the exposure conditionusing the exposure condition setting unit 409 in a manner that a mainsubject portion will have a predetermined luminance level (appropriateluminance level). The system control unit 408 outputs a command signalto the drive unit 406. According to the command signal, the drive unit406 is driven to appropriately adjust the light amount according to thedetermined exposure condition. The system control unit 408 also controlsthe entire imaging apparatus 400 according to the user settings in theoperation unit 407.

The exposure condition setting unit 409 is included in the systemcontrol unit 408. When, for example, a preview image of a scene isformed before an image of the scene is actually captured, the exposurecondition setting unit 409 temporarily reads RGB images corresponding toone screen (data of RGB images corresponding to one screen formed usingRGB image signals), which are output from the A/D converter 405, andtemporarily stores the RGB images into an internal memory (not shown).The exposure condition setting unit 409 then calculates the exposurecondition in a manner that an output value corresponding to a mainportion of the subject P2 will have an appropriate luminance level. Forexample, the exposure condition of the imaging apparatus 400 is set in amanner that a person's face portion of the image will have a luminancelevel of 70% when the dynamic range of an output image is 100%, which isthe dynamic range of output image signals with values corresponding toluminance levels of 0 to 100%.

The image processing unit 410 includes a dynamic-range increasing unit411 and a dynamic-range compression unit 412. The dynamic-rangeincreasing unit 411 increases the dynamic range of RGB image signalsoutput from the A/D converter 405. The dynamic-range compression unit412 compresses the dynamic range of the increased-dynamic-range image,which is an image whose dynamic range has been increased by thedynamic-range increasing unit 411, to a predetermined dynamic range. Theimage processing unit 410 further includes a signal processing unit 413,which processes signals to have a signal format suitable for the displayunit 414 and the storage unit 415 (for example, NTSC, JPEG, or MPEG).

The display unit 414 is arranged, for example, on the back surface ofthe main body of the imaging apparatus 400. The display unit 414includes a display, such as a color LCD. The display unit 414 displaysan image that is formed using the image signals processed by the imageprocessing unit 410 on the display, such as the color LCD.

The storage unit 415 stores the image (image signals) processed by thesignal processing unit 413. The storage unit 415 may be formed by, forexample, a recording medium known in the art, such as a hard disk drive(HDD) or a semiconductor memory.

4.2 Operation of the Imaging Apparatus

The main operation of the imaging apparatus 400 of the presentembodiment will now be described in the chronological order.

As shown in FIG. 32A, the drive unit 406 outputs control signals (B) forcontrolling the image sensor 404 and the image processing unit 410 basedon synchronization signals (A) having cycles T. As described later, theexposure condition for the predetermined exposure time t₀, which isindicated by control signals (C), is set in a manner that a main portionof the subject P2 will have an appropriate luminance level. When, forexample, the main portion is a person's face portion, the exposurecondition of the imaging apparatus 400 is set in a manner that the mainportion (person's face portion) will have a luminance level of 70% withrespect to an output dynamic range of 100%. The aperture 403 adjusts theamount of light incident on the image sensor 404 based on the setexposure condition. The range of outputs of the A/D converter 405 thatare not saturated is set with a margin. More specifically, the A/Dconverter 405 can have outputs corresponding to luminance levels up to150% without saturating the outputs. In other words, among image signalsoutput from the A/D converter 405, image signals with valuescorresponding to luminance levels up to 150% are not saturated.

When capturing a still image, the imaging apparatus 400 performs theexposure processing of one cycle, which is the processing performed inthe cycle T (frame cycle), only once. When capturing a moving image, theimaging apparatus 400 repeatedly performs the exposure processing of onecycle a plurality of times as required according to the time for whichthe imaging apparatus is being operated to capture the image.

In the present embodiment, the exposure of one cycle is assumed to bedivided into a plurality of sequential divisional exposures. The imagingapparatus 400 is assumed to obtain a plurality of images through theplurality of sequential divisional exposures. The images obtainedthrough the divisional exposures are assumed to be images correspondingto sub-frames. Before converted by A/D conversion, the sub-frame imagesare referred to “sub-frames analogue images”, or simply “analogueimages”. After converted by A/D conversion, the sub-frame images arereferred to as “sub-frame digital images”, or simply “digital images”.Image signals that are used to form the sub-frame analogue images arereferred to as “analogue image signals”, whereas image signals that areused to form the sub-frame digital images are referred to as “digitalimage signals”. The same notation applies both to the processing forcapturing a moving image and the processing for capturing a still image.

As shown in FIG. 32B, to increase the output margin of the A/D converter405 further, the drive unit 406 drives the image sensor 404 in a mannerthat the image sensor 404 is subjected to the sequential divisionalexposures, each of which is performed for the time t₁, which is ¼ of thepredetermined exposure time t₀. The predetermined exposure time t₀ ist₀=4*t₁. The drive unit 406 drives the image sensor 404 and controls thetime for which the image sensor 404 is exposed using, for example, theelectronic shutter function of the image sensor 404.

As indicated by control signals (B), an analogue image (analogue imagesignals), is obtained through each divisional exposure performed for theexposure time t₁. The analogue image is then converted to a digitalimage (digital image signals) by the A/D converter 405 during the timet₂, which is indicated by control signals (D). The digital image signalis transferred to the image processing unit 410.

The image processing unit 410 stores digital images transferred from theA/D converter 405 into memory units (image storage units). The imageprocessing unit 401 reads image data of the same pixels of the differentdigital images from the same addresses of the different memory units.The image processing unit 401 increases the dynamic range of each pixel,and adds up the image data of the same pixels of the different digitalimages. As a result, the image processing unit 401 obtains adynamic-range-increased image 102, which is an image with an increaseddynamic range, as indicated by signals (E).

The analogue image obtained through each divisional exposure that isperformed for the exposure time t₁, which is shorter than thepredetermined exposure time t₀, has a smaller output value (smallervalues of analogue image signals) than an analogue image that would beobtained through an exposure performed for the predetermined exposuretime t₀. Thus, an image resulting from A/D conversion of the an analogueimage obtained through the divisional exposure will have a larger marginwith respect to a saturation limit of an output of the A/D converter.The four analogue images obtained by the image sensor 404 through thefour divisional exposures are converted by the A/D converter 405 togenerate four digital images. The digital images are then subjected tothe dynamic range increase, and are added up to form an image. Thisimage has the same output value as an image that would be formed basedon an image obtained through the exposure performed for thepredetermined exposure time t₀. As a result, the image formed by addingup the images obtained through the divisional exposures has the S/Nratio equivalent to the S/N ratio of the image that would be formedbased on the image obtained through the exposure performed for thepredetermined exposure time t₀.

The imaging apparatus 400 is further advantageous in that the imagingapparatus 400 forms an increased-dynamic-range image, which is an imagewith an increased dynamic range, by converting the four analogue imagesobtained through the divisional exposures, each of which is performedfor the exposure time t₁, to four digital images and increasing thedynamic range of each pixels of the four digital images and then addingup the same pixels of the four digital images.

In other words, the exposure light amount used in the divisionalexposure performed for the exposure time t₁ is smaller than the exposurelight amount that would be used in the exposure performed for thepredetermined exposure time t₀. Therefore, the output value of eachanalogue image obtained through the divisional exposure, which is outputfrom the image sensor 404, is smaller accordingly. As a result, theoutput margin of the A/D converter 405 with respect to its saturationlimit is t₀/t₁ times greater. Therefore, the output resulting from theA/D conversion is less likely to be saturated.

Alternatively, the analogue images obtained by the image sensor 404 maybe converted to digital images before the analogue images are subjectedto the dynamic range increase, and the digital images may be subjectedto the dynamic range increase. More specifically, the imaging apparatus400 may first obtain a plurality of analogue images by subjecting theimage sensor 404 to a plurality of sequential divisional exposures,which are performed to prevent an output of the A/D converter 405 frombeing saturated. The imaging apparatus 400 may then convert the analogueimages to digital images using the A/D converter 405, and increase thedynamic range of each pixel of the different digital images and add upthe same pixels of the different digital images. The imaging apparatus400 with this structure also obtains a dynamic-range-increased image,which is an image with an increased dynamic range, for the same reasonsas described above.

4.2.1 Details of the Dynamic-Range Increasing Unit 411

The dynamic-range increasing unit 411 included in the imaging apparatus400 of the present invention will now be described in detail withreference to FIGS. 33 to 35.

FIG. 33 is a block diagram showing the structure of the dynamic-rangeincreasing unit 411. FIG. 34 is a diagram describing the characteristicof the dynamic-range increasing unit 411 included in the imagingapparatus 400 of the present embodiment. FIG. 35 is a diagram describingthe imaging state of the subject P2, whose image is captured by theimaging apparatus 400 of the present embodiment.

As shown in FIG. 33, the dynamic-range increasing unit 411 includesimage storage units 431 a to 431 d and an addition unit 430. The imagestorage units 431 a to 431 d store images (including R, G, and Bimages), which are output from the A/D converter 415. The addition unit430 adds up the same pixels of different images stored in the imagestorage units 431 a to 431 d after increasing the dynamic range of eachpixel.

To enable the dynamic range increase, the addition unit 430 may beformed by, for example, an adder with increased bits. When, for example,the output of the A/D converter 405 is 10-bit data and the exposure timeis divided by four (four divisional exposures are performed), theaddition unit 430 may be formed by an adder with 12 bits, which has a2-bit increase to the output bit number of the A/D converter 405. Thenumber of bit increase should not be limited to 2 bits. The number ofbit increase is determined in advance according to the number by whichthe exposure time is divided (in other words, the number of divisionalexposures). When, for example, the exposure time is divided by eight,the required bit increase would be 3 bits or more. In the same manner,when the exposure time is divided by sixteen, the required bit increasewould be 4 bits or more, and when the exposure time is divided by two,the bit increase would be 1 bit or more.

FIG. 34 is a graph showing the dynamic-range conversion characteristicof the dynamic-range increasing unit 411.

As shown in FIG. 34, when, for example, the amount of incident lightused in the exposure performed for the predetermined exposure time t₀corresponds to a luminance level of 150%, the aperture 403 adjusts thelight amount in a manner that an output of the A/D converter 405 willhave a luminance level corresponding to the saturation limit (aluminance level of 150%). In this case, the exposure time t₁ is t₁=t₀/4.

Thus, when the exposure performed for the predetermined exposure time t₀and the divisional exposure performed for the exposure time t₁ arecompared under the same incident light amount condition, an analogueimage (analogue image signals) obtained by the image sensor 404 throughthe divisional exposure has a smaller output value, which corresponds toa luminance level of 37.5%. As a result, the output resulting from theA/D conversion (see a straight line L105 indicating the input and outputcharacteristic in FIG. 34) has a smaller signal value, which correspondsto a luminance level of 37.5%. In this manner, when the divisionalexposure is performed for the exposure time t₁, the output margin of theA/D converter 405 with respect to the saturation limit is 4 timesgreater. With the divisional exposure performed for the exposure timet₁, the imaging apparatus 400 prevents the output of the A/D converter405 from being saturated until the incident light amount reaches anamount corresponding to a luminance level of 600% (4*150%). When, forexample, the luminance level of a person's face portion of the scene isadjusted to 70%, a sky and cloud portion of the scene will have aluminance level of 500%. In this case, the output with the increaseddynamic range is not saturated.

Moreover, the image storage units 431 a to 431 c store the digitalimages resulting from A/D conversion performed by the A/D converter 405.The addition unit 430 increases the dynamic range of each pixel of thedifferent images stored in the image storage units 431 a to 431 c andadds up the same pixels of the images. As a result, the addition unit430 forms a dynamic-range-increased image. In this manner, the imagingapparatus 400 obtains a dynamic-range-increased image whose dynamicrange has been increased by the addition unit 430 to values ranging from0 to 600% (see a straight line L102 indicating the input and outputcharacteristic in FIG. 34).

4.2.2 Advantageous Effects of the Dynamic Range Increase

The advantageous effects of the dynamic range increase will now bedescribed with reference to FIG. 35.

For an outdoor subject scene, the incident light amount varies greatlydepending on portions of the subject. For example, an analogue image 106of a backlit subject P2 has a low luminance level in a face vicinityportion 108 of the subject P2. The luminance level of the face vicinityportion 108 of the subject P2 may be corrected to 70% through thedynamic range increase. In this case, a bright portion of the scene,which is for example a sky and cloud portion 107, will normally have aluminance level of 500% or more. When the A/D converter 405 has anoutput saturation limit of 150%, the imaging apparatus would form animage in which the tone levels of the sky and cloud portion 107 aresaturated, and would fail to appropriately reproduce the sky and cloudportion 107.

In contrast, the imaging apparatus 400 of the present embodiment usesthe dynamic-range increasing unit 411 to increase the dynamic range ofeach digital image signal, which is obtained by A/D conversion, to 0 to600% without causing saturation, and obtains digital image signals whosevalues are not saturated. In this case, the tone levels of a sky andcloud portion (portion that normally has a luminance level of 500% ormore) of an image formed by the imaging apparatus 400 are not saturated.As a result, the imaging apparatus 400 appropriately reproduces the skyand cloud portion with natural tone levels.

4.2.3 Details of the Dynamic-Range Compression Unit 412

The dynamic-range compression unit 412 included in the imaging apparatus400 will now be described in detail with reference to FIGS. 35 and 36.FIG. 36 is a diagram describing the characteristics of the dynamic-rangecompression unit 412.

The dynamic-range compression unit 412 compresses the increased dynamicrange of a signal (signal whose tone level has been converted based onthe input and output characteristic line 102 in FIG. 34) to a dynamicrange of 100% or less. In other words, the dynamic-range compressionunit 412 compresses the dynamic range of a large input (large inputsignal value) to generate an output (output value) with a reduceddynamic range. In the dynamic range compression, the dynamic-rangecompression unit 412 compresses the dynamic range of a large input(large signal value) corresponding to a bright portion of the subject P2with a larger compression ratio while maintaining an appropriateluminance level of a face vicinity portion 108 of the subject P2. Forexample, the dynamic-range compression unit 412 performs the dynamicrange compression in a manner that an output corresponding to the facevicinity portion 108 of the subject P2 will have a value correspondingto an appropriate luminance level, or for example, a luminance level of70%. In the present embodiment, the dynamic range of the output iscompressed linearly up to a point S (output of 80%) according to astraight line 103 indicating a linear conversion characteristic in FIG.36. With the linear conversion characteristic line 103, the dynamicrange of the output corresponding to the face vicinity portion 108 andsimilar portions (with a luminance level of 70%) is changed linearly(the tone levels of pixels with luminance levels of about 70% areconverted linearly). The dynamic range of the output is compressed witha gradient of a characteristic curve 104, which is kinked at the point Sand extends to the point R (curve with a knee), from the point S to apoint Q in FIG. 36. As indicated by the characteristic curve 104, theincreased dynamic range of 600% of the image signal is compressed to100%.

The dynamic-range compression unit 412 may not use the curve having theknee described above but may use other dynamic range compressionmethods. The other dynamic range compression methods will be describedlater.

As described above, the imaging apparatus 400 first increases thedynamic range of an image signal and then compresses the increaseddynamic range of the image signal to a predetermined dynamic range. Theimaging apparatus 400 obtains an analogue image signal that does notsaturate an output of the A/D converter 405 even when capturing an imageof a subject, such as an outdoor subject, whose input light amountsgreatly vary depending on portions of the subject. As a result, theimaging apparatus 400 reduces the output range of signals (range ofvalues of its obtained (output) digital image signals) both for a brightportion and a dark portion of the subject to an appropriate dynamicrange.

4.2.4 Operation of the Imaging Apparatus

The operation of the imaging apparatus 400 according to the presentembodiment will now be described with reference to a flowchart shown inFIG. 37.

In step S100, the aperture 403 adjusts the light amount in a manner thata main portion of the subject will have an appropriate luminance levelthrough exposures of the image sensor 404 whose total exposure time isequal to the predetermined exposure time t₀.

In step S102, the image sensor 404 is subjected to four sequentialdivisional exposures, each of which is performed for the same exposuretime. The total exposure time of the four divisional exposures is equalto the predetermined exposure time t₀. The image sensor 404 obtains fouranalogue images through the four divisional exposures.

In step S104, the A/D converter 405 converts the four analogue images byA/D conversion to generate four digital images.

In step S106, the dynamic-range increasing unit 411 increases thedynamic range of each pixel of the four different digital images andadds up the same pixels of the different digital images. As a result,the dynamic-range increasing unit 411 forms a dynamic-range-increasedimage 102, which is an image with an increased dynamic range.

As described above, the imaging apparatus 400 performs the processing instep S100 before, for example, performing a plurality of divisionalexposures. Through the processing in step S100, the imaging apparatus400 adjusts the light amount according to the predetermined exposuretime t₀ in a manner that a person's face portion of an image will havean appropriate luminance level. The imaging apparatus 400 then performsthe processing in step S102 to S106 to obtain a dynamic-range-increasedimage.

Alternatively, the processing in steps S100 to S106 may be changed inthe following manner.

The image sensor 404 is driven in a manner that the image sensor 404forms four analogue images through four divisional short exposures (witha high shutter speed), which are performed to prevent an output of theA/D converter 405 from being saturated. The A/D converter 405 thenconverts the four analogue images to four digital images. The dynamicrange of each pixel of the four different images is then increased, andthe same pixels of the different images are added up to form adynamic-range-increased image.

Through the processing described above, the imaging apparatus 400obtains a dynamic-range-increased image while preventing the S/N ratioof the image from deteriorating.

Modifications

Modifications of the dynamic range compression will now be describedwith reference to FIGS. 38 to 40.

The dynamic-range compression unit 412 described above performs thedynamic range compression based on the nonlinear tone curve with thecharacteristics shown in FIG. 36. The dynamic-range compression unit 412with this dynamic range compression enables a main portion of an image,such as a face portion, to have an appropriate luminance level, and tonelevels of highlight portions with a wide range of luminance levels to beretained without saturation while effectively using the large dynamicrange of 600%, which has been increased by the dynamic-range increasingunit 411.

However, as shown in FIG. 36, the dynamic-range compression unit 412with the dynamic range compression conversion characteristic (tone levelconversion characteristic) does not compress the dynamic range of tonelevels corresponding to luminance levels of 0 to 80%, and compresses thedynamic range of tone levels corresponding to luminance levels exceeding80% and up to 600% to the dynamic range of 80 to 100%. The dynamic-rangecompression unit 412 with this dynamic range compression conversioncharacteristic can retain tone levels of highlight portions with a widerange of luminance levels. Although the dynamic-range compression unit412 retains tone levels of a highlight portion, the gradient of theinput and the output (the gradient of the dynamic range compressionconversion characteristic curve) will be extremely small in thehighlight portion. Consequently, the contrast will be extremely small inthe highlight portion.

The gradient of the input and output characteristic curve of the dynamicrange compression of the highlight portion decreases more as thedynamic-range increasing unit 411 increases the dynamic range of thehighlight portion by a greater degree. As a result, although thehighlight portion retains its changing tone levels through the tonelevel conversion, the tone values (signal values) of the highlightportion differ only too slightly from one another. In this case, theimage captured by the imaging apparatus differs insignificantly from asaturated image captured with a conventional technique (the imagecaptured by the imaging apparatus of the present embodiment can be seenas a saturated image). This can be the fundamental problem of thedynamic range compression of the imaging apparatus. Despite thisproblem, the imaging apparatus of the present invention is stillobviously advantageous over the saturation occurring with theconventional technique (the phenomenon in which a highlight portion of acaptured image is saturated), because the gradient of the input andoutput characteristic curve of the dynamic range compression of ahighlight portion of a scene would not be extremely small when thedynamic range compression performed with the dynamic range compressionconversion characteristic shown in FIG. 36 is applied to the sceneportions with luminance levels of several hundred percent or less.

To solve this fundamental problem, the tone characteristic may bechanged according to positions of the tone levels on an image.International Publication No. WO 2005/027041 describes one suchtechnique. More specifically, the patent document describes a techniquefor using a vicinity portion of processing target pixels of an image.With this technique, for example, the histogram of a vicinity portion ofprocessing target pixels of an image is measured, and the tone curveused for the target pixels is determined based on the distribution ofvalues of the histogram. Alternatively, the average luminance level ofthe vicinity portion of the processing target pixels is calculated, andthe tone curve used for the target pixels is determined according to theluminance level of the vicinity portion.

When the technique described above is applied to the dynamic-rangecompression unit 412, the imaging apparatus 400 will solve the problemof low contrast in the highlight portion by minimizing the contrastdecrease caused by the dynamic range compression. More specifically, theimaging apparatus 400 converts large dynamic range image information,which is information about an image with the increased large dynamicrange generated by the dynamic-range increasing unit 411, to smalldynamic range information, which is information about an image with adynamic range of 100% or less, without causing almost no loss of theimage information. Consequently, the imaging apparatus 400 maximizes theeffects of the dynamic range increase of the present invention.

Dynamic Range Compression Using Visualization Characteristics

The dynamic range compression performed by the dynamic-range compressionunit 412 using the above technique will now be described with referenceto FIG. 38. FIG. 38 is a block diagram showing the structure of thedynamic-range compression unit 412.

As shown in FIG. 38, the dynamic-range compression unit 412 includes avicinity luminance detection unit 4121 and a dynamic tone correctionunit 4122. The vicinity luminance detection unit 4121 detects arepresentative value (for example, an average value) of luminance levelsof pixels that are in the vicinity of processing target pixels. Thedynamic tone correction unit 4122 has a dynamic range compression curvethat changes according to an output of the vicinity luminance detectionunit 4121.

FIG. 39 is a diagram describing the operation of the dynamic-rangecompression unit 412.

The dynamic-range compression unit 412 has five tone curves (tone levelconversion characteristic curves shown in FIG. 39). The dynamic-rangecompression unit 412 selects one of the five tone curves according tothe luminance level of a vicinity portion of processing target pixels.As shown in FIG. 39, the dynamic-range compression unit 412 selects thecurve a when the vicinity portion of the target pixels is the darkest,the curve c when the vicinity portion of the target pixels is thebrightest, and the curve b when the vicinity portion of the targetpixels has an intermediate luminance level between the darkest and thebrightest cases. Although the present embodiment describes the case inwhich the dynamic-range compression unit 412 has the five tone levelconversion characteristic curves for ease of explanation, the number ofthe characteristic curves should not be limited to five. It ispreferable that the dynamic-range compression unit 412 actually uses inthe tone level conversion of image signals as many tone level conversioncharacteristic curves as curves that can be assumed substantiallycontinuous to one another. For example, the dynamic-range compressionunit 412 may have several tens of tone level conversion characteristiccurves.

The dynamic-range compression unit 412 uses the tone level conversioncharacteristic curve a when processing image signals corresponding to adark vicinity portion of target pixels included in an image (forexample, the face vicinity portion 108 in FIG. 35). Thus, the tonelevels of image signals corresponding to the person's face portion areconverted to tone levels (signal values) corresponding to an appropriateluminance value of about 70%. The dynamic-range compression unit 412uses the tone level conversion characteristic curve c when processingimage signals corresponding to an extremely bright vicinity portion oftarget pixels included in the image (for example, the sky and cloudportion 107 in FIG. 35). Thus, image signals corresponding to the skyand cloud portion 107 are converted to image signals with a sufficientlylarge number of tone levels and a sufficiently high contrast. Throughthe dynamic range compression performed by the dynamic-range compressionunit 412, the imaging apparatus of the present embodiment forms an imagewith a high definition (image signals) that is seen natural.

The dynamic range compression technique described above is based on thevisual characteristics of humans. Such a technique is called avisualization-characteristic-based technique (visual processing). Thetechnique is based on the visual characteristics of the eyes of humans.The human eyes increase sensitivity when viewing a bright portion anddecrease sensitivity when viewing a dark portion. The imaging apparatusof the present embodiment uses the visualization-characteristic-basedtechnique to effectively compress the large dynamic range of an imageand prevent an image from being seen unnatural. As a result, the imagingapparatus of the present embodiment obtains a natural image.

The imaging apparatus of the fourth embodiment compresses an increaseddynamic range of 1000% or more of an increased-dynamic-range image, orfor example, an increased dynamic range of 3200% of anincreased-dynamic-range image, which is obtained by the dynamic-rangeincreasing unit 412, to a dynamic range or 100% or less without visuallydegrading the increased large dynamic range of the image.

4.3 Advantageous Effects of the Imaging Apparatus

The advantageous effects of the imaging apparatus of the presentembodiment will now be described using images that are actually capturedby the imaging apparatus (camera) of the present embodiment and aconventional imaging apparatus (camera) with reference to FIGS. 40A and40B.

FIG. 40A shows an image captured through conventional camera processingwhose exposure is controlled in a manner that a person's face, which isa main subject, has an appropriate level of luminance. FIG. 40B shows animage captured by the imaging apparatus 400 that includes thedynamic-range compression unit 412.

The images shown in FIGS. 40A and 40B have almost the same luminancelevel in dark portions, such as the shaded portion of the waterwheel,and also have almost the same luminance level in the face portion.However, the images have different luminance levels in bright portions,such as the sky and cloud portion and the person's clothing portion inwhich sunlight is reflected. In detail, the bright portions of the imageshown in FIG. 40A are saturated, and overexposed and fail toappropriately reproduce color. In contrast, the bright portions of theimage shown in FIG. 40B are seen natural as the bright portions of theimage retain their continuously changing tone levels as well as thecontours and the contrast of the sky and the cloud.

The images in FIGS. 40A and 40B also differ from each other in the colorof the sky. The sky portion of the image in FIG. 40A is overexposed andfails to appropriately reproduce color, whereas the sky portion of theimage in FIG. 40B appropriately reproduces color of the blue sky. Thecomparison between the two images reveals that the dynamic-rangecompression unit 412 of the present embodiment effectively compressesthe increased dynamic range of image signals to the small dynamic rangeof 100% and obtains a natural image while effectively using the largedynamic range increased by the dynamic-range increasing unit 411.

As described above, the imaging apparatus 400 of the present embodimentfirst adjusts the light amount in a manner that a main portion of thesubject will have an appropriate luminance level through exposures whosetotal exposure time is equal to the predetermined exposure time t₀, andthen obtains four analogue signals by subjecting the image sensor 404 tofour divisional exposures, each of which is performed for the exposuretime t₁, which is ¼ of the predetermined exposure time t₀. Thus, theimaging apparatus 400 reduces the output value of each analogue image to¼, and increases the output margin of the A/D converter 405 four times.To increase the luminance level of the dark images, the imagingapparatus 400 increases the dynamic range of each pixel of the fourdifferent images and adds up the same pixels of the different images. Asa result, the imaging apparatus 400 forms a dynamic-range-increasedimage. The luminance level of the dynamic-range-increased image is ashigh as the luminance level of an image that would be formed based on animage obtained through the exposure performed for the predeterminedexposure time t₀. As a result, the imaging apparatus 400 obtains thedynamic-range-increased image while preventing the S/N ratio of theimage from deteriorating.

The imaging apparatus 400 further uses the dynamic-range-compressionunit 412 to compress the increased dynamic range of image signalsobtained by the dynamic-range increasing unit 411 to a dynamic range of100% or less with the tone characteristic of the image signals changingaccording to positions on the image. The dynamic range compression unit412 compresses an increased dynamic range of 1000% or more of anincreased-dynamic-range image, such as an increased dynamic range of1200%, to a dynamic range of 100% or less without visually degrading thelarge dynamic range of the image.

Although the above embodiment describes the case in which thepredetermined exposure time t₀ is divided by four, and four divisionalexposures are performed each for the exposure times t₁, the number bywhich the predetermined exposure time t₀ is divided should not belimited to four. For example, the predetermined exposure time t₀ may bedivided by two, or sixteen, according to the required ratio of thedynamic range increase. The number by which the predetermined exposuretime t₀ is divided may be increased or decreased as required. When thepredetermined exposure time t₀ is divided by two, the dynamic rangeincrease ratio would be two times (300%). When the predeterminedexposure time t₀ is divided by sixteen, the dynamic range increase ratiowould be 16 times (2400%). In this manner, the number by which thepredetermined exposure time t₀ is divided may be increased or decreasedaccording to a required dynamic range. When the predetermined exposuretime t₀ is divided by N (where N is a natural number), the dynamic rangeincrease ratio would be N times. In this case, the output can beincreased to 0 to N times (the output can be increased up to thesaturation limit (%) of the output of the A/D converter 405). It is onlyrequired that the imaging apparatus 400 divide the predeterminedexposure time t₀ by at least two and perform the processing describedabove. In this case, the imaging apparatus 400 has the advantageouseffects described above.

Fifth Embodiment

An imaging apparatus according to a fifth embodiment of the presentinvention will now be described with reference to FIGS. 41 to 43.

The imaging apparatus according to the fifth embodiment improves imagequality by correcting an image shift between images caused by movementof the imaging apparatus due to hand shake of the user. A method used bythe imaging apparatus of the fifth embodiment will now be described.

The imaging apparatus according to the fifth embodiment differs from theimaging apparatus 400 of the fourth embodiment only in its dynamic-rangeincreasing unit 511 shown in FIG. 41, which replaces the dynamic-rangeincreasing unit 411. The components of the imaging apparatus of thefifth embodiment that are the same as the components of the imagingapparatus 400 of the fourth embodiment are given the same referencenumerals as those components, and will not be described.

The dynamic-range increasing unit 511 included in the imaging apparatusof the fifth embodiment will be described with reference to FIGS. 41 to43.

FIG. 41 is a block diagram showing the structure of the dynamic-rangeincreasing unit 511 included in the imaging apparatus of the presentembodiment. FIGS. 42A and 42B are diagrams describing apparatus movementcorrection. FIG. 43 is a diagram describing apparatus-movement detectionunits 542 to 544.

The dynamic-range increasing unit 411 included in the imaging apparatus400 of the fourth embodiment receives four images of the subject P2,which have been obtained through a plurality of divisional exposures,and increases the dynamic range of each pixel of the four differentmages and adds up the sane pixels of the different images to form adynamic-range-increased image. The dynamic-range increasing unit 511 ofthe present embodiment detects and corrects an image shift betweenimages that is caused by movement of the imaging apparatus due to handshake of the user who operates the imaging apparatus (hereafter referredto as “apparatus movement”) before increasing the dynamic range of eachpixel of the four different images and adding up the pixels of thedifferent images. Consequently, the imaging apparatus 400 obtains animage with reduced image shifts caused by apparatus movement.

The dynamic-range increasing unit 511 includes image storage units 531 ato 531 d, apparatus-movement detection units 542 to 544, anapparatus-movement determination unit 545, and an addition unit 541. Theimage storage units 531 a to 531 d store images (including R, G, and Bimages), which are output from the A/D converter 405. Theapparatus-movement detection units 542 to 544 detect the degree anddirection of an image shift that can be caused by movement of theimaging apparatus due to hand shake of the user based on the imagesstored in the image storage units 531 a to 531 d. The apparatus-movementdetermination unit 545 determines whether apparatus movement hasoccurred based on the degree of an image shift. When theapparatus-movement determination unit 545 determines that apparatusmovement has occurred, the addition unit 541 calculates a correctionvalue that is used to eliminate an image shift caused by the apparatusmovement. The addition unit 541 shifts images according to the timingsof a control signal based on the calculated correction value and adds upthe images.

The apparatus-movement detection unit 542 compares two images stored inthe image storage units 531 a and 531 b, and detects the degree anddirection of an image shift between the two images. In the same manner,the apparatus-movement detection unit 543 compares two images stored inthe image storage units 531 b and 531 c, and detects the degree anddirection of an image shift between the two images. Theapparatus-movement detection unit 543 compares two images stored in theimage storage units 531 c and 531 d, and detects the degree anddirection of an image shift between the two images.

When the apparatus-movement determination unit 545 determines that thetwo images stored in the image storage units 531 a and 531 b has animage shift, the addition unit 541 calculates a correction value basedon the degree and direction of the image shift detected by theapparatus-movement detection unit 542. In the same manner, when the twoimages stored in the image storage units 531 b and 531 c has an imageshift, the addition unit 541 calculates a correction value based on thedegree and direction of the image shift detected by theapparatus-movement detection unit 543. When the two images stored in theimage storage units 531 c and 531 d has an image shift, the additionunit 541 calculates a correction value based not only on the degree anddirection of the image shift detected by the apparatus-movementdetection unit 543 but also on the degree and direction of the imageshift detected by the apparatus-movement detection unit 544.

The apparatus movement correction will now be described with referenceto FIGS. 42A and 42B.

Vertical image shifts caused by apparatus movement and horizontal imageshifts caused by apparatus movement are corrected through the sameprocessing. For ease of explanation, only vertical image shiftcorrection will be described below.

FIG. 42A shows images obtained through divisional exposures performed attimings t−1 to t+2. Timing t is the reference timing. At timing t−1, aframe image immediately preceding a frame image (image B) obtained attiming t is obtained. At timing t+1, a frame image immediately followingthe frame image obtained at timing t is obtained. The same applies totimings t+N and t−N (where N is any natural number).

It is preferable to use, as the reference image (image B in FIGS. 42Aand 42B), an image obtained at the timing close to the timing at whichthe imaging apparatus is operated to capture an image (the timing whenthe shutter is released). It is preferable that the image storage units531 a to 531 d of the dynamic-range increasing unit 511 storesequentially obtained images in a manner that the images are in thechronological order in the image storage units. It is preferable to usean intermediate image among the sequentially obtained images as thereference image. As shown in FIGS. 42A and 42B, the image B at timing tis used as the reference image. An image shift of each of the images A,C, and D with respect to a reference H of the image B will be described.

As shown in FIG. 42A, the image A obtained at timing t−1 is shifteddownward by an amount Δh1 with respect to the image B. In the samemanner, the image D obtained at timing t+2 is shifted upward by anamount Δh2 with respect to the image B. The image C obtained at timingt+1 is not shifted with respect to the image B.

As shown in FIG. 42B, when the images have an image shift caused byapparatus movement, the apparatus-movement detection unit 542 detects andownward image shift amount Δh1 based on comparison between the image Band the image A. The apparatus-movement determination unit 545 thendetermines whether apparatus movement has occurred based on whether theimage shift amount is greater than a predetermined value C. When theapparatus-movement determination unit 545 determines that apparatusmovement has occurred (Δh1>C), the addition unit 541 calculates thecoordinates of the image A to which a correction value (Δh1) forcorrecting the shift of the coordinates of the image A has been added,and uses the values of pixels at the calculated coordinates when addingup the pixels of the different images.

In the same manner, the apparatus-movement detection unit 543 comparesthe image B and the image C. When the apparatus-movement determinationunit 545 determines that apparatus movement has not occurred (the imageshift amount=0<C), the addition unit 541 uses the values of pixels ofthe image C that are at the same coordinates as the coordinates of theimage B when adding up the pixels of the different images.

The apparatus-movement detection unit 544 compares the image C and theimage D, and detects an upward image shift amount of Δh2. Theapparatus-movement determination unit 545 determines whether apparatusmovement has occurred based on whether the image shift amount is greaterthan the predetermined value C.

When the apparatus-movement determination unit 545 determines thatapparatus movement has occurred (Δh2>C), the addition unit 541calculates the coordinates of the image D from which a correction value(Δh2+0) for correcting the shift of the coordinates of the image D hasbeen subtracted, and uses the values of pixels at the calculatedcoordinates when adding up the pixels of the different images. Whencorrecting the image shift of the image D, the addition unit 541calculates the coordinates of the image D to which the image shift ofthe image C with respect to the image B has also been added. Theaddition unit 541 then uses the values of pixels at the calculatedcoordinates when adding up the pixels of the different images. Theaddition unit 541 does not use the values of target pixels that are outof range when adding the pixels of the different images.

Through such apparatus movement correction, the addition unit 541eliminates the image shifts of the images A, C, and D with respect tothe image B before increasing the dynamic range of each pixel of thedifferent images A, C, and D and adding up the pixels of the differentimages. This correction eliminates image shifts caused by movement ofthe imaging apparatus of the present embodiment due to hand shake of theuser.

As shown in FIG. 43, when comparing the images A and B stored in theimage storage units 531 a and 531 b, the apparatus-movement detectionunit 542 may set predetermined areas 545 a to 545 f in the image A, andcompare the predetermined areas 545 a to 545 f of the image A with thecorresponding predetermined areas 546 a to 546 f of the image B. Theapparatus-movement detection unit 542 may then calculate the average ofvalues Δh_(ave) 1 to Δh_(ave) 6 indicating the degree and direction ofimage shifts obtained through comparisons between the areas, and use thecalculated average as a correction value used by the addition unit 541.When the calculated average is a value with decimals, interpolation maybe performed between values of the target pixels and values of pixelsadjacent to the target pixels in the vertical, horizontal, and diagonaldirections according to the decimal part of the value. Alternatively,the value with decimals may be rounded to the nearest whole number, andthe rounded value may be used as the correction value. The use of thesepredetermined areas reduces the computation load for the comparisonbetween the images, and shortens the processing time required for thecomparison and also reduces the scale of required hardware.

The apparatus-movement detection units 543 and 544 perform the sameprocessing as the apparatus-movement detection unit 532.

As described above, the imaging apparatus of the present embodiment usesthe dynamic-range increasing unit 511 to eliminate image shifts ofimages caused by movement of the imaging apparatus due to hand shake ofthe user before increasing the dynamic range of each pixel of thedifferent images A to D and adding up the pixels of the different imagesA to D. In addition to the advantages effects of the imaging apparatus400 of the fourth embodiment, the imaging apparatus of the presentembodiment has the advantageous effect of obtaining adynamic-range-increased image with reduced image shifts.

Moreover, the imaging apparatus of the present embodiment obtainsanalogue images by subjecting the image sensor 404 to four divisionalexposures. Thus, the exposure time for each exposure through which oneanalogue image is obtained is reduced to ¼, and therefore an image shiftcaused by apparatus movement due to hand shake of the user is alsoreduced to ¼. In the same manner, an image shift caused by movement ofthe subject is also reduced to ¼. The imaging apparatus of the presentembodiment then converts the four analogue images to digital images byA/D conversion performed by the A/D converter 405, and shifts images ina direction to eliminate image shifts caused by apparatus movementaccording to the degree and direction of the image shifts detected bythe apparatus-movement detection units 542 to 544 and adds up the imagesin which the image shifts have been eliminated. Therefore, the imagingapparatus of the present embodiment obtains an image with smaller imageshifts caused by apparatus movement as compared with an image formedwith a conventional technique that does not divide the exposure into aplurality of divisional exposures.

Although the above embodiment describes the case in which an image shiftis detected by comparing corresponding pixels of a plurality of images,the present invention should not be limited to this structure. Theimaging apparatus of the present embodiment may detect the degree anddirection of apparatus movement using, for example, a gyroscope, and mayshift images in a direction to eliminate an image shift caused by theapparatus movement according to the degree and direction of theapparatus movement, and then increase the dynamic range of the eachpixel of the different images A to D and add up the pixels of thedifferent images A to D.

Alternatively, the apparatus-movement detection units 542 to 544 maydetect the degree and direction of an image shift based on G elements ofthe images A to D.

The addition unit 541 may assume that an image shift has a value of 0when no degree of image shift is detected by the apparatus-movementdetection units 542 to 544. In this case, the apparatus-movementdetermination unit 545 may be eliminated. In this case, the additionunit 541 is simply required to shift images in a direction to eliminateimage shifts according to the degree and direction of image shiftsdetected by the apparatus-movement detection units 542 to 544 and add upthe images.

Sixth Embodiment

An imaging apparatus according to a sixth embodiment of the presentinvention will now be described with reference to FIGS. 44 to 45.

The imaging apparatus according to the sixth embodiment differs from theimaging apparatus 400 of the fourth embodiment only in its dynamic-rangeincreasing unit 611 shown in FIG. 44, which replaces the dynamic-rangeincreasing unit 411. The components of the imaging apparatus of thesixth embodiment that are the same as the components of the imagingapparatus 400 of the fourth embodiment are given the same referencenumerals as those components, and will not be described.

FIG. 44 is a block diagram showing the structure of the dynamic-rangeincreasing unit 611 included in the imaging apparatus of the presentembodiment. FIGS. 45A and 45B are diagrams describing apparatus movementcorrection and subject movement correction.

The dynamic-range increasing unit 511 included in the imaging apparatusof the fifth embodiment shifts images in a direction to eliminate imageshifts caused by apparatus movement due to hand shake of the user andobtains an image with reduced image shifts caused by apparatus movement.The dynamic-range increasing unit 611 of the present embodiment alsoprevents an image shift caused by movement of the subject P2 (hereafterreferred to as “subject movement”) in addition to an image shift causedby apparatus movement.

As shown in FIG. 44, the dynamic-range increasing unit 611 includesimage storage units 631 a to 631 d, apparatus-movement detection units642 to 644, and an apparatus-movement determination unit 645. The imagestorage units 631 a to 631 d store RGB images, which are output from theA/D converter 405. The apparatus-movement detection units 642 to 644detect an image shift caused by apparatus movement based on the imagesstored in the image storage units 631 a to 631 d. The apparatus-movementdetermination unit 645 determines whether apparatus movement hasoccurred based on the degree of an image shift. The dynamic-rangeincreasing unit 611 further includes subject-movement detection units652 to 654 and an addition unit 651. The subject-movement detectionunits 652 to 654 detect whether subject movement has occurred in a localarea by comparing the same pixels of two different images stored in theimage storage units 631 a to 631 d. The addition unit 651 calculates acorrection value that is used to eliminate an image shift when theapparatus-movement detection units 642 to 644 detect apparatus movement.The addition unit 651 shifts the image signals using the correctionvalue and then adds up the image signals. When the subject-movementdetection units 652 to 654 detect subject movement, the addition unit651 excludes pixels having the detected subject movement from the pixelsof the image signals that are added up.

The subject-movement detection unit 652 compares the same pixels of twodifferent images stored in the image storage units 631 a and 631 b aftershifting the images according to the degree and direction of an imageshift detected by the apparatus-movement detection unit 642, and detectspixels having subject movement. In the same manner, the subject-movementdetection unit 653 compares the same pixels of two different imagesstored in the image storage units 631 b and 631 c after shifting theimages according to the degree and direction of an image shift detectedby the apparatus-movement detection unit 643, and detects pixels havingsubject movement. The subject-movement detection unit 654 compares thesame pixels of two different images stored in the image storage units631 c and 631 d after shifting the images according to the degree anddirection of an image shift detected by the apparatus-movement detectionunit 644, and detects pixels having subject movement.

The apparatus movement correction and the subject movement correctionwill now be described with reference to FIGS. 45A and 45B.

Vertical image shifts caused by apparatus movement and subject movementand horizontal image shifts caused by apparatus movement and subjectmovement are corrected through the same processing. For ease ofexplanation, only vertical image shift correction will be describedbelow.

FIG. 45A shows images obtained through divisional exposures performed attimings t−1 to t+2. An image shift of each of the images A, C, and Dwith respect to a reference H of the image B will be described.

As shown in FIG. 45A, the image A obtained at timing t−1 is shifteddownward by an amount Δh1 with respect to the image B. In the samemanner, the image D obtained at timing t+2 is shifted upward by anamount Δh2 with respect to the image B. The image C obtained at timingt+1 is not shifted with respect to the image B.

As shown in FIG. 45B, when images have an image shift caused byapparatus movement, the dynamic-range increasing unit 611 corrects theapparatus movement by eliminating the image shift of the four images,which have been obtained through the divisional exposures, based on thecorrection value calculated according to the degree and direction of theimage shift detected by the apparatus-movement detection units 642 to644, in the same manner as the dynamic-range increasing unit 611 of thefifth embodiment. The subject-movement detection units 652 to 654 thencompare the same pixels of selected adjacent two of the images A to D,and detect whether pixels in a certain area have subject movement. Theaddition unit 651 then adds up the pixels of the different images A to Din a manner determined based on whether apparatus movement and subjectmovement have been detected. In FIG. 45B, subject movement is detectedin areas X and Y.

A local movement of the subject P2 may not be detected by directlycomparing the same pixels of two sub-frame images and detecting adifference between the two sub-frame images, but may be detected bycomparing larger areas of the two images including target pixels afterthe images are processed through a low-pass filter. Alternatively, alocal motion of the subject P2 may be detected by comparing pixels ofthe two images through pattern matching.

The addition unit 651 obtains the coordinates (coordinates of positionson the image) of a local area having subject movement (for example,areas X and Y) from the subject-movement detection units 652 to 654. Theaddition unit 651 excludes pixels included in the areas X and Y havingsubject movement, from which image shifts caused by apparatus movementhave been eliminated, from the pixels of the different images that areadded up. The addition unit 651 uses the pixels included in areas havingno subject movement, from which image shifts caused by apparatusmovement have been eliminated, as the pixels of the images that areadded up.

When the addition unit 651 have excluded some pixels from the pixels ofthe images that are added up, the addition unit 651 adjusts theluminance level of an image formed by adding up the images according tothe number of images added. For example, pixels of the images A and Dcorresponding to the local area are assumed to have subject movement asshown in FIG. 45B. For the local area, the addition unit 651 uses thepixels of only the images B and C. In this case, the luminance level ofthe local area of the image formed by adding up the images B and C wouldbe 2/4 of the luminance level of an image that would be formed by addingup the four images. In this case, the addition unit 651 increases theluminance level of the local area of the image 4/2 times. In the samemanner, the luminance level of an image that is formed by adding upthree images would be 3/4 of the luminance level of an image that wouldbe formed by adding up the four images. In this case, the addition unit651 increases the luminance level of the local area of the image 4/3times. This luminance level adjustment may not be performed by theaddition unit 651, and may be performed by, for example, the signalprocessing unit 413.

In this manner, the imaging apparatus adds up the pixels of the fourimages corresponding to image areas other than the areas X and Y havingsubject movement after increasing the dynamic range of each pixel. Inthis manner, the imaging apparatus obtains image signals with no imageshifts caused by apparatus movement and subject movement.

As described above, the imaging apparatus of the present embodiment usesthe dynamic-range increasing unit 611 to perform the processingconsidering subject movement. In addition to the advantages effects ofthe imaging apparatus 500 of the fifth embodiment including thedynamic-range increasing unit 511, the imaging apparatus of the presentembodiment has the advantageous effect of obtaining image signals inwhich image shifts caused by subject movement have been furthereliminated.

Moreover, the imaging apparatus of the present embodiment divides theexposure time by four. In this case, each digital image resulting fromA/D conversion will have the same degree of image shift caused byapparatus movement. Thus, image shifts caused by apparatus movement areeliminated effectively without the need to consider differences in imageshift amounts due to different exposure times. In this manner, theimaging apparatus of the present embodiment subjects the image sensor404 to short exposures (with a high shutter speed) and obtains an imagewith reduced image shifts caused by apparatus movement and subjectmovement.

The imaging apparatus of the present embodiment detects subject movementafter detecting apparatus movement by comparing images and shifting theimages in a direction to eliminate an image shift between the imagescaused by apparatus movement. However, the present invention should notbe limited to such a structure. When, for example, the images areconsidered to have no image shifts caused by apparatus movement, such aswhen the imaging apparatus is fixed on a stage like a tripod, theimaging apparatus of the present embodiment may only detect subjectmovement by comparing images without apparatus movement correction.

The imaging apparatus of the present embodiment may optically correctapparatus movement, and may detect subject movement in the mannerdescribed above in a plurality of images in which image shifts caused byapparatus movement have been eliminated in advance. The imagingapparatus with this structure has the same advantageous effects asdescribed above. In this case, the imaging apparatus of the presentembodiment effectively uses a plurality of images in which image shiftscaused by apparatus movement have been eliminated in advance. Theoptical correction of the apparatus movement eliminates the need to setareas for apparatus movement detection within the imaging area of theimage sensor 404, and enables the imaging area of the image sensor 404to be used effectively.

Seventh Embodiment

An imaging apparatus according to a seventh embodiment of the presentinvention will now be described with reference to FIGS. 46 to 47.

The imaging apparatus according to the seventh embodiment differs fromthe imaging apparatus 400 of the fourth embodiment only in itsdynamic-range increasing unit 711 shown in FIG. 46, which replaces thedynamic-range increasing unit 411. The components of the imagingapparatus of the seventh embodiment that are the same as the componentsof the imaging apparatus 400 of the fourth embodiment are given the samereference numerals as those components, and will not be described.

FIG. 46 is a block diagram showing the structure of the dynamic-rangeincreasing unit 711 included in the imaging apparatus of the presentembodiment. FIGS. 47A and 47B are diagrams describing apparatus movementcorrection and subject movement correction.

The dynamic-range increasing unit 611 included in the imaging apparatusof the sixth embodiment performs the apparatus movement correction ofcorrecting an image shift caused by apparatus movement, and performs thesubject movement correction of eliminating an image shift caused bysubject movement. With such subject movement correction, image signalscorresponding to a local area having subject movement are excluded fromimage signals that are added up after the dynamic range of the signalsis increased. The dynamic-range increasing unit 711 of the presentembodiment does not exclude image signals corresponding to a local areahaving subject movement, and obtains image signals not only with reducedimage shifts caused by apparatus movement but also with reduced imageshifts caused by subject movement.

As shown in FIG. 46, the dynamic-range increasing unit 711 includesimage storage units 731 a to 731 d, apparatus-movement detection units742 to 744, and an apparatus-movement determination unit 745. The imagestorage units 731 a to 731 d store RGB images, which are output from theA/D converter 405. The apparatus-movement detection units 742 to 744detect an image shift caused by apparatus movement based on the imagesstored in the image storage units 731 a to 731 d. The apparatus-movementdetermination unit 745 determines whether apparatus movement hasoccurred based on the degree of an image shift. The dynamic-rangeincreasing unit 711 further includes subject-movement detection units752 to 754 and coordinate conversion units 762 to 764. Thesubject-movement detection units 752 to 754 detect whether subjectmovement has occurred in a local area by comparing the same pixels oftwo different images stored in the image storage units 731 a to 731 d.The coordinate conversion units 762 to 764 convert the coordinates ofpixels included in a local area having subject movement in a manner toeliminate subject movement. The dynamic-range increasing unit 711further includes an addition unit 761. The addition unit 761 calculatesa correction value that is used to eliminate an image shift caused byapparatus movement when the apparatus-movement detection units 742 to744 detect the apparatus movement, and adds up image signals aftershifting the image signals based on the correction value. When subjectmovement is detected, the addition unit 761 may use the pixels in whichthe subject movement has been eliminated, which are output from thecoordinate conversion units 762 to 764, as the pixels of the images thatare added up.

The dynamic-range increasing unit 711 further includes asubject-movement determination unit 755. When an image shift betweenimages caused by subject movement has been detected, thesubject-movement determination unit 755 detects a difference betweenpixels of the images corresponding to a local area in which the subjectmovement has been detected, and determines whether the subject movementcan be eliminated by the coordinate conversion units 762 to 764.

When the subject-movement determination unit 755 determines that thesubject movement cannot be eliminated, the addition unit 761 excludesthe pixels of the images corresponding to the local area from the pixelsof the images that are added up. When the subject-movement determinationunit 755 determines that the subject movement can be eliminated, theaddition unit 761 uses the pixels of the images corresponding to thelocal area in which the subject movement has been eliminated, which areoutput from the coordinate conversion units 762 to 764, as the pixels ofthe images that are added up.

As described above, the imaging apparatus of the present embodiment usesthe subject-movement determination unit 755 included in thedynamic-range increasing unit 711 to determine a difference betweenpixels of images corresponding to a local area in which subject movementhas been detected. Thus, the imaging apparatus appropriately handles anarea in which a part of the subject cannot be detected. When, forexample, a hand of the subject is hidden behind the subject, the imagingapparatus excludes pixels of images corresponding to an area includingthe hand from the pixels of the images that area added up.

The coordinate conversion unit 762 shifts images according to the degreeand direction of an image shift detected by the apparatus-movementdetection unit 742, and then converts the coordinates of the pixels ofimages signals stored in the image storage unit 731 a for which subjectmovement has been detected. In the same manner, the coordinateconversion unit 763 shifts images according to the degree and directionof an image shift detected by the apparatus-movement detection unit 743,and then converts the coordinates of the pixels of images signals storedin the image storage unit 731 c for which subject movement has beendetected. In the same manner, the coordinate conversion unit 764 shiftsimages according to the degree and direction of an image shift detectedby the apparatus-movement detection unit 744, and then converts thecoordinates of the pixels of images signals stored in the image storageunit 731 d for which subject movement has been detected.

The apparatus movement correction and the subject movement correctionwill now be described with reference to FIGS. 47A and 47B.

Vertical image shifts caused by apparatus movement and subject movementand horizontal image shifts caused by apparatus movement and subjectmovement are corrected through the same processing. For ease ofexplanation, only vertical image shift correction will be describedbelow.

As shown in FIG. 47A, the image A obtained at timing t−1 is shifteddownward by an amount Δh1 with respect to the image B. In the samemanner, the image D obtained at timing t+2 is shifted upward by anamount Δh2 with respect to the image B. The image C obtained at timingt+1 is not shifted with respect to the image B.

As shown in FIG. 47B, when images have an image shift caused byapparatus movement, the dynamic-range increasing unit 60 corrects theapparatus movement by eliminating the image shift of the four images ofthe subject P2, which have been obtained through the divisionalexposures, based on the correction value according to the degree anddirection of the image shift detected by the apparatus-movementdetection units 742 to 744, in the same manner as the dynamic-rangeincreasing unit 511 of the fifth embodiment. The subject-movementdetection units 752 to 754 then compare the same pixels of selectedadjacent two of the images A to D, and detect pixels in an area havingsubject movement. The addition unit adds up the pixels of the images Ato D in a manner determined according to whether apparatus movement andsubject movement have been detected. In FIG. 47B, subject movement isdetected in areas X and Y.

The coordinate conversion units 762 to 764 move and rotate each pixel ofthe images corresponding to a local area in which subject movement hasbeen detected by the subject-movement detection units 752 to 754 (areasX and Y) by affine transformation, and outputs signals, whose subjectmovement has been corrected, to the addition unit 761.

When the subject-movement determination unit 755 determines that subjectmovement cannot be corrected by the coordinate conversion units 762 to764, the addition unit 761 excludes the pixels at the calculatedcoordinates from the pixels of the images that are added up. Thesubject-movement determination unit 755 may determine that subjectmovement cannot be corrected when, for example, a hand of the subject P2is hidden behind the subject P2 and the hand cannot be detected, orwhen, for example, an arm or a face of the subject P2 is partiallyhidden behind an obstacle. In this manner, when certain pixels of imagesare expected not to match one another even if the coordinate conversionunits 762 to 764 shift images by converting the coordinates of thepixels, the addition unit 761 excludes the pixels of the images at thecalculated coordinates from the pixels of the images that are added up.

As described above, the imaging apparatus of the present embodiment hasthe advantageous effect of eliminating subject movement of pixels of theimages corresponding to the local area before increasing the dynamicrange of each pixel and adding up the pixels of the images A to D, inaddition to the advantages effects of the imaging apparatus of the sixthembodiment including the dynamic-range increasing unit 611. The imagingapparatus of the present embodiment obtains a dynamic-range-increasedimage with reduced image shifts caused by apparatus movement and subjectmovement. The imaging apparatus of the present embodiment adjusts theluminance level of an image formed by adding up the pixels of the imagesaccording to the number of images added, in the same manner as in thesixth embodiment.

Other Embodiments

For simplicity and easy understanding of the present invention, thefourth to seventh embodiments of the present invention do not describevideo gamma processing, which is included in normal camera processing.Video gamma processing is not essential to the present invention. Thepresent invention should not be limited to the structure without videogamma processing, whereas the structure with video gamma processing alsohas the same advantageous effects as the present invention.

The functional blocks described in the above embodiments may beincorporated with other camera signal processing functions and may beimplemented by hardware using for example an integrated circuit, or maybe implemented as incorporated software using a CPU of an integratedcircuit. Alternatively, the functional blocks may be implemented byindependent computer application software. The various functionsdescribed above may be implemented by both software and hardware.

When the functions described above are implemented by hardware, thefunctions described in each of the above embodiments may be implementedby separate integrated circuits, or some or all of the functions may beimplemented by a single-chip integrated circuit. For example, anintegrated circuit may function as an imaging apparatus including theoptical system 416, the image sensor 404, the A/D converter 405, and alight amount adjustment unit that adjusts the amount of light accordingto the predetermined exposure time t₀ in a manner that a main portion ofthe subject will have an appropriate luminance level. In this case, theintegrated circuit includes the drive unit 406 and the dynamic-rangeincreasing unit 411 (or 511, 611, or 711). As described above, the driveunit 406 drives the image sensor 404 in a manner that the image sensor404 obtains four analogue images through four sequential divisionalexposures, each of which is performed for the same exposure time. Thetotal exposure time of the four divisional exposures is equal to thepredetermined exposure time t₀. The dynamic-range increasing unit 411increases the dynamic range of each pixel of four different digitalimages, which have been obtained by converting the four analogue imagesby A/D conversion performed by the A/D converter 405, and adds up thesame pixels of the different images. As a result, the integrated circuitforms a dynamic-range-increased image, which is an image with anincreased dynamic range, and outputs the dynamic-range increased image.

Alternatively, the drive unit 406 may drive the image sensor 404 in amanner that the image sensor 404 obtains four analogue images throughfour sequential divisional exposures, which are performed to prevent anoutput of the A/D converter 405 from being saturated, and thedynamic-range increasing unit 411 (or 511, 611, or 711) may increase thedynamic range of each pixel of four different digital images, which aresignals resulting from conversion of the four analogue images performedby the A/D converter 405, and add up the same pixels of the differentimages. As a result, the integrated circuit may form adynamic-range-increased image, which is an image with an increaseddynamic range, and output the dynamic-range-increased image.

In this case, the integrated circuit obtains large dynamic range imagesignals while preventing the S/N ratio of the signals fromdeteriorating. The number of divisional exposures into which theexposure is divided should not be limited to four. The exposure may bedivided into N divisional exposures (N>1, where N is a natural number)according to a required dynamic range. In this case, the integratedcircuit obtains N images through the N divisional exposures, andincreases the dynamic range of each pixel of the images and adds up thepixels of the images.

The integrated circuit should not be limited to an LSI (large scaleintegrated) circuit. The integrated circuit may be referred to as an IC(integrated circuit), a system LSI, a super LSI circuit, or an ultra LSIdepending on the degree of integration of the circuit.

The integrated circuit may be formed by a dedicated circuit or ageneral-purpose processor. For example, the integrated circuit may be anFPGA (field programmable gate array), which is an integrated circuitprogrammable after the semiconductor chip is manufactured, or areconfigurable processor, which is an integrated circuit in whichinternal circuit cells are reconfigurable or more specifically internalcircuit cells can be reconnected or reset.

If any circuit integration technology emerges as an advancement of thesemiconductor technology or as a derivative of the semiconductortechnology, the technology may be used to integrate the functionalblocks of the imaging apparatus. For example, biocomputers may becomeapplicable with the advancement of biotechnology.

The application software may not be stored in a recording medium, suchas a disk, when distributed. Alternatively, the software may bedownloaded via a network.

Although the above embodiments describe the apparatus movementcorrection and the subject movement correction only for vertical imageshifts, the present invention should not be limited to vertical imageshift correction. Image shifts in other direction, such as horizontalimage shifts and rotational image shifts may be corrected through thesame processing as described above.

Although the above embodiments describe the case in which the processingis performed in units of frames, the present invention should not belimited to such processing. For example, the processing may be performedin units of fields.

Although the above embodiments describe the case in which the imagestorage units (431 a to 431 d, 531 a to 531 d, 631 a to 631 d, 731 a to731 d) are separate functional blocks, the present invention should notbe limited to such a structure. Alternatively, a single memory may bedivided based on addresses and may function as the plurality of imagestorage units.

Although the above embodiments describe the case in which the imagingapparatus of the present invention is applied to a digital camera forcapturing still images, the present invention should not be limited tosuch application. The present invention is also applicable to a videocamera for capturing moving images. The present invention is applicableto a video camera without requiring any changes to the embodimentsdescribed above except that the mechanical shutter of the digital camerais replaced by an electronic shutter of the video camera.

The imaging apparatus according to each of the above embodiments of thepresent invention is applicable to a still camera for capturing an imageof a still subject, a digital camera, such as a video camera, forcapturing an image of a moving subject, a monitoring camera formonitoring a subject, a mobile telephone having the imaging function, aninformation device having the imaging function, and an integratedcircuit that functions as an imaging apparatus.

The structures described in detail in the above embodiments are mereexamples of the present invention, and may be changed and modifiedvariously without departing from the scope and spirit of the invention.

APPENDIXES

The present invention may also be expressed as follows.

First Appendixes

Appendix 1

An imaging apparatus for electronically capturing an image of a subject,the apparatus comprising: an optical system having a light amountadjusting function; an image sensor operable to read an optical imagethat is formed by the optical system; an A/D converter operable toconvert an output of the image sensor by analogue-to-digital conversion;and a dynamic-range increasing unit operable to linearly increase adynamic range of an output of the A/D converter.

Appendix 2

An imaging apparatus for electronically capturing an image of a subject,the apparatus comprising: an optical system having a light amountadjusting function; an image sensor operable to read an optical imagethat is formed by the optical system; an A/D converter operable toconvert an output of the image sensor by analogue-to-digital conversion;a dynamic-range increasing unit operable to linearly increase a dynamicrange of an output of the A/D converter; and a dynamic-range compressionunit operable to nonlinearly compress a dynamic range of an output ofthe dynamic-range increasing unit to a predetermined dynamic range.

Appendix 3

The imaging apparatus according to one of appendixes 1 and 2, whereinthe dynamic-range increasing unit includes an amplifier unit operable todigitally increase sensitivity of an input, and the dynamic-rangeincreasing unit linearly increases the dynamic range by allowing anoutput of the amplifier unit to pass without saturating the output ofthe amplifier unit.

Appendix 4

The imaging apparatus according to one of appendixes 2 and 3, whereinthe dynamic-range compression unit nonlinearly compresses the dynamicrange using a conversion characteristic that changes according to aluminance level of a brightest portion of an image.

Appendix 5

The imaging apparatus according to one of appendixes 2, 3, and 4,wherein the dynamic-range compression unit compresses the dynamic rangeusing a conversion characteristic that changes according to a spatialposition on the image.

Appendix 6

The imaging apparatus according to one of appendixes 2, 3, and 4,wherein the dynamic-range compression unit compresses the dynamic rangeusing a conversion characteristic that changes according to a luminancelevel of a vicinity portion of a processing target pixel.

Appendix 7

An imaging method for electronically capturing an image of a subject,the method comprising: setting an exposure condition of an image sensorin a manner that a highlight of a scene is not saturated; converting anoutput of the image sensor to digital image data; linearly increasing adynamic range of the digital image data in a manner that a main subjectof the image data has a predetermined luminance level; and nonlinearlycompressing the increased dynamic range to a predetermined dynamicrange.

Appendix 8

The imaging method according to appendix 7, wherein the dynamic range iscompressed using a conversion characteristic that changes according to alevel of a brightest portion in the image data in the nonlinear dynamiccompression step.

Appendix 9

The imaging method according to appendix 7, wherein the dynamic range iscompressed using a conversion characteristic that changes according to aspatial position in the image data in the nonlinear dynamic rangecompression step.

Appendix 10

An integrated circuit used in an imaging apparatus including an opticalsystem having a light amount adjusting function, an image sensor, and anA/D converter, the integrated circuit comprising: a dynamic-rangeincreasing unit operable to linearly increase a dynamic range of anoutput of the A/D converter; and a dynamic-range compression unitoperable to nonlinearly compress a dynamic range of an output of thedynamic-range increasing unit to a predetermined dynamic range.

Appendix 11

The integrated circuit according to appendix 10, wherein thedynamic-range increasing unit includes an amplifier unit operable todigitally increase sensitivity of an input, and the dynamic-rangeincreasing unit linearly increases the dynamic range by allowing anoutput of the amplifier unit to pass without saturating the output ofthe amplifier unit.

Appendix 12

The integrated circuit according to one of appendixes 10 and 11, whereinthe dynamic-range compression unit nonlinearly compresses the dynamicrange using a conversion characteristic that changes according to aluminance level of a brightest portion of an image.

Appendix 13

The integrated circuit according to one of appendixes 10, 11, and 12,wherein the dynamic-range compression unit compresses the dynamic rangeusing a conversion characteristic that changes according to a spatialposition on the image.

Appendix 14

A program used in an imaging apparatus including an optical systemhaving a light amount adjusting function, an image sensor, and an A/Dconverter, the program comprising: converting an output of the imagesensor to digital image data; linearly increasing a dynamic range of thedigital image data in a manner that a main subject of the image data hasa predetermined luminance level; and nonlinearly compressing theincreased dynamic range to a predetermined dynamic range.

Second Appendixes

Appendix 1

An imaging apparatus comprising: an optical unit having a light amountadjusting function; an image sensor operable to read an optical imagethat is formed by the optical unit; an A/D converter operable to convertan analogue image output from the image sensor to a digital image; adrive unit operable to drive the image sensor in a manner that the imagesensor obtains a plurality of analogue images through a plurality ofsequential divisional exposures each of which is performed for anidentical exposure time, a total exposure time of the divisionalexposures being equal to a predetermined exposure time; and adynamic-range increasing unit operable to increase a dynamic range ofcorresponding pixels of a plurality of digital images resulting fromconversion of the plurality of analogue images performed by the A/Dconverter and add up the corresponding pixels of the digital images toform an image with an increased dynamic range, and output the image withthe increased dynamic range.

Appendix 2

The imaging apparatus according to appendix 1, further comprising: adynamic-range compression unit operable to nonlinearly compress theincreased dynamic range of the image to a predetermined dynamic range.

Appendix 3

The imaging apparatus according to one of appendixes 1 and 2, whereinthe dynamic-range increasing unit has an output dynamic range that is atleast greater than an input dynamic range.

Appendix 4

The imaging apparatus according to appendix 2, wherein the dynamic-rangecompression unit compresses the increased dynamic range of the imageusing a conversion characteristic that changes according to a spatialposition on the image.

Appendix 5

The imaging apparatus according to one of appendixes 1 to 4, wherein,before the image sensor is subjected to the plurality of divisionalexposures, an amount of light is adjusted in a manner that a mainportion of the subject will have a predetermined luminance level throughan exposure performed for the predetermined exposure time.

Appendix 6

The imaging apparatus according to one of appendixes 1 to 5, furthercomprising: an apparatus-movement detection unit operable to detect adegree and a direction of an image shift caused by movement of theimaging apparatus between two different digital images among theplurality of digital images, wherein the dynamic-range increasing unitshifts the digital images in a direction to eliminate the image shiftaccording to the degree and the direction of the image shift detected bythe apparatus-movement detection unit, and add up the correspondingpixels of the digital images.

Appendix 7

The imaging apparatus according to appendix 6, further comprising: asubject-movement detection unit operable to detect movement of thesubject in a local area on the digital images by comparing correspondingpixels of the two different digital images in which the image shiftcaused by the movement of the imaging apparatus has been eliminated,wherein when the movement of the subject is detected by thesubject-movement detection unit, the dynamic-range increasing unitexcludes the corresponding pixels of the different digital images havingthe movement of the subject from the pixels of the digital images thatare added up.

Appendix 8

The imaging apparatus according to one of appendixes 1 to 5, furthercomprising: a subject-movement detection unit operable to detectmovement of the subject in a local area on two different digital imagesamong the plurality of digital images by comparing corresponding pixelsof the two different digital images, wherein when the movement of thesubject is detected by the subject-movement detection unit, thedynamic-range increasing unit excludes the corresponding pixels of thedifferent digital images having the movement of the subject from thepixels of the digital images that are added up.

Appendix 9

The imaging apparatus according to one of appendixes 1 to 5, furthercomprising: a subject-movement detection unit operable to obtain aplurality of digital images in which movement of the imaging apparatushas been corrected, and detect movement of the subject in a local areaon the digital images by comparing corresponding pixels of two differentdigital images among the plurality of digital images, wherein when themovement of the subject is detected by the subject-movement detectionunit, the dynamic-range increasing unit excludes the correspondingpixels of the different digital images having the movement of thesubject from the pixels of the digital images that are added up.

Appendix 10

The imaging apparatus according to appendix 9, wherein movement of theimaging apparatus is corrected by means of optical correction.

Appendix 11

The imaging apparatus according to appendix 6, further comprising: asubject-movement detection unit operable to detect movement of thesubject in a local area on the digital images by comparing correspondingpixels of the two different digital images in which the image shiftcaused by the movement of the imaging apparatus has been eliminated; anda coordinate conversion unit operable to convert coordinates ofcorresponding pixels of different digital images having the movement ofthe subject detected by the subject-movement detection unit to eliminatethe movement of the subject, wherein the dynamic-range increasing unituses, as the pixels of the digital images that are added up, pixels thatare at coordinates obtained by the coordinate conversion unit throughconversion of the coordinates of the pixels.

Appendix 12

The imaging apparatus according to appendix 11, further comprising: asubject-movement determination unit operable to determine whether themovement of the subject will be eliminated by converting the coordinatesof the pixels, wherein when the subject-movement determination unitdetermines that the movement of the subject will not be eliminated byconverting the coordinates of the pixels, the dynamic-range increasingunit excludes the pixels having the movement of the subject from thepixels of the digital images that are added up.

Appendix 13

An imaging apparatus comprising: an optical unit having a light amountadjusting function; an image sensor operable to read an optical image ofa subject that is formed by the optical unit; an A/D converter operableto convert an analogue image that is output from the image sensor to adigital image; a drive unit operable to drive the image sensor in amanner that the image sensor obtains a plurality of analogue imagesthrough a plurality of sequential divisional exposures that areperformed to prevent an output of the A/D converter from beingsaturated; and a dynamic-range increasing unit operable to increase adynamic range of corresponding pixels of a plurality of digital imagesresulting from conversion of the plurality of analogue images performedby the A/D converter and add up the corresponding pixels of the digitalimages to form an image with an increased dynamic range, and output theimage with the increased dynamic range.

Appendix 14

An imaging apparatus comprising: an optical unit having a light amountadjusting function; an image sensor operable to read an optical image ofa subject that is formed by the optical unit; an A/D converter operableto convert an analogue image that is output from the image sensor to adigital image; a drive unit operable to drive the image sensor in amanner that the image sensor obtains a plurality of analogue imagesthrough a plurality of sequential divisional exposures; asubject-movement detection unit operable to detect movement of thesubject in a local area on a plurality of digital images resulting fromconversion of the plurality of analogue images performed by the A/Dconverter by comparing corresponding pixels of two different digitalimages among the plurality of digital images; and an addition unitoperable to add up corresponding pixels of the plurality of digitalimages obtained by the A/D converter, wherein when the movement of thesubject is detected by the subject-movement detection unit, the additionunit excludes the corresponding pixels of the different digital imageshaving the movement of the subject from the pixels of the digital imagesthat are added up.

Appendix 15

A digital camera comprising: the imaging apparatus according to one ofappendixes 1 to 14.

Appendix 16

An integrated circuit used in an imaging apparatus including an opticalunit having a light amount adjusting function, an image sensor operableto read an optical image of a subject that is formed by the opticalunit, an A/D converter operable to convert an analogue image that isoutput from the image sensor to a digital image, and a light amountadjustment unit operable to adjust an amount of light according to apredetermined exposure time in a manner that a main portion of thesubject has a predetermined luminance level, the integrated circuitcomprising: a drive unit operable to drive the image sensor in a mannerthat the image sensor obtains a plurality of analogue images through aplurality of sequential divisional exposures each of which is performedfor an identical exposure time, a total exposure time of the divisionalexposures being equal to the predetermined exposure time; and adynamic-range increasing unit operable to increase a dynamic range ofcorresponding pixels of a plurality of digital images resulting fromconversion of the plurality of analogue images performed by the A/Dconverter and add up the corresponding pixels of the images to form animage with an increased dynamic range, and output the image with theincreased dynamic range.

Appendix 17

An imaging method used in an imaging apparatus including an optical unithaving a light amount adjusting function, an image sensor operable toread an optical image of a subject that is formed by the optical unit,an A/D converter operable to convert an analogue image that is outputfrom the image sensor to a digital image, and a drive unit operable todrive the image sensor, the method comprising: adjusting an amount oflight according to a predetermined exposure time in a manner that a mainportion of the subject has a predetermined luminance level; driving theimage sensor in a manner that the image sensor obtains a plurality ofanalogue images through a plurality of sequential divisional exposureseach of which is performed for an identical exposure time, a totalexposure time of the divisional exposures being equal to thepredetermined exposure time; and increasing a dynamic range ofcorresponding pixels of a plurality of digital images resulting fromconversion of the plurality of analogue images performed by the A/Dconverter and adding up the corresponding pixels of the images to forman image with an increased dynamic range, and outputting the image withthe increased dynamic range.

Appendix 18

The imaging method according to appendix 17, further comprising:nonlinearly compressing the increased dynamic range of the image to apredetermined dynamic range.

Appendix 19

The imaging method according to one of appendixes 17 and 18, wherein inthe dynamic-range increasing step, an output dynamic range is at leastgreater than an input dynamic range.

Appendix 20

The imaging method according to appendix 18, wherein in thedynamic-range compression step, the increased dynamic range of the imageis compressed using a conversion characteristic that changes accordingto a spatial position on the image.

Appendix 21

The imaging method according to one of appendixes 17 to 20, furthercomprising: detecting a degree and a direction of an image shift causedby movement of the imaging apparatus between two different digitalimages among the plurality of digital images, wherein in thedynamic-range increasing step, the digital images are shifted in adirection to eliminate the image shift according to the degree and thedirection of the image shift detected in the apparatus-movementdetection step, and the corresponding pixels of the digital images areadded up.

Appendix 22

The imaging method according to appendix 21, further comprising:detecting movement of the subject in a local area on the digital imagesby comparing corresponding pixels of the two different digital images inwhich the image shift caused by the movement of the imaging apparatushas been eliminated, wherein when the movement of the subject isdetected, the corresponding pixels of the different digital imageshaving the movement of the subject are excluded from the pixels of thedigital images that are added up in the dynamic-range increasing step.

Appendix 23

The imaging method according to one of appendixes 17 to 20, furthercomprising: detecting movement of the subject in a local area on twodifferent digital images among the plurality of digital images bycomparing corresponding pixels of the two different digital images,wherein when the movement of the subject is detected, the correspondingpixels of the different digital images having the movement of thesubject are excluded from the pixels of the digital images that areadded up in the dynamic-range increasing step.

Appendix 24

The imaging method according to one of appendixes 17 to 20, furthercomprising: obtaining a plurality of digital images in which movement ofthe imaging apparatus has been corrected, and detecting movement of thesubject in a local area on the digital images by comparing correspondingpixels of two different digital images among the plurality of digitalimages, wherein when the movement of the subject is detected in thesubject-movement detection step, the corresponding pixels of thedifferent digital images having the movement of the subject are excludedfrom the pixels of the digital images that are added up in thedynamic-range increasing step.

Appendix 25

The imaging method according to appendix 24, wherein movement of theimaging apparatus is corrected by means of optical correction.

Appendix 26

The imaging method according to appendix 21, further comprising:detecting movement of the subject in a local area on the digital imagesby comparing corresponding pixels of the two different digital images inwhich the image shift caused by the movement of the imaging apparatushas been eliminated; and converting coordinates of corresponding pixelsof different digital images having the movement of the subject detectedin the subject-movement detection step to eliminate the movement of thesubject, wherein pixels that are at coordinates obtained in thecoordinate conversion step through conversion of the coordinates of thepixels are used as the pixels of the digital images that are added up inthe dynamic-range increasing step.

Appendix 27

The imaging method according to appendix 26, further comprising:determining whether the movement of the subject will be eliminated byconverting the coordinates of the pixels, wherein when thesubject-movement determination step determines that the movement of thesubject will not be eliminated by converting the coordinates of thepixels, the pixels having the movement of the subject are excluded fromthe pixels of the digital images that are added up in the dynamic-rangeincreasing step.

Appendix 28

An imaging method used in an imaging apparatus including an optical unithaving a light amount adjusting function, an image sensor operable toread an optical image of a subject that is formed by the optical unit,an A/D converter operable to convert an analogue image that is outputfrom the image sensor to a digital image, and a drive unit operable todrive the image sensor, the method comprising: driving the image sensorin a manner that the image sensor obtains a plurality of analogue imagesthrough a plurality of sequential divisional exposures that areperformed to prevent an output of the A/D converter from beingsaturated; and increasing a dynamic range of corresponding pixels of aplurality of digital images resulting from conversion of the pluralityof analogue images performed by the A/D converter and adding up thecorresponding pixels of the digital images to form an image with anincreased dynamic range, and outputting the image with the increaseddynamic range.

Appendix 29

An imaging method used in an imaging apparatus including an optical unithaving a light amount adjusting function, an image sensor operable toread an optical image of a subject that is formed by the optical unit,and an A/D converter operable to convert an analogue image that isoutput from the image sensor to a digital image, the method comprising:driving the image sensor in a manner that the image sensor obtains aplurality of analogue images through a plurality of sequentialdivisional exposures; detecting movement of the subject in a local areaon a plurality of digital images resulting from conversion of theplurality of analogue images performed by the A/D converter by comparingcorresponding pixels of two different digital images among the pluralityof digital images; and adding up corresponding pixels of the pluralityof digital images obtained by the A/D converter, wherein when themovement of the subject is detected in the subject-movement detectionstep, the corresponding pixels of the different digital images havingthe movement of the subject are excluded from the pixels of the digitalimages that are added up in the addition step.

Industrial Applicability

The imaging apparatus, the imaging method, the integrated circuit thatfunctions as the imaging apparatus, and the program used in the imagingapparatus of the present invention appropriately capture a large dynamicrange image that exists in the natural world as an image that is seennatural without saturation on a conventional display having a smalldynamic range, and prevent the S/N ratio of the image fromdeteriorating. Thus, the imaging apparatus, the imaging method, theintegrated circuit, and the program of the present invention areapplicable not only to digital still cameras, but also to other imagingapparatuses that capture images, such as video cameras, built-in camerasof mobile telephones, monitoring cameras, security cameras, and camerasincorporated in the eyes of robots, and are also applicable toapplication software, such as RAW converter software and photoretouching software for digital still cameras.

What is claimed is:
 1. A visual processing apparatus comprising: aspatial processor that receives an image signal and outputs arepresentative value of luminance levels of a vicinity portion of atarget pixel; and a visual processor that receives the image signal andthe representative value, obtains an output signal using a conversioncharacteristic, and outputs the output signal, wherein the conversioncharacteristic is represented by a first tone curve, the first tonecurve has an input variable corresponding to the image signal and anoutput variable corresponding to the output signal, the first tone curvechanges according to the representative value, the first tone curve hasa property in which a rate of change of the output variable with respectto the input variable is greater than that of a second tone curve, inthe vicinity of a point of intersection between the first tone curve andthe second tone curve, and the second tone curve represents arelationship between the input variable and the output variableoutputted when the image signal is equal to the representative value. 2.A visual processing method comprising: receiving an image signal andoutputting a representative value of luminance levels of a vicinityportion of a target pixel; and receiving the image signal and therepresentative value, obtaining an output signal using a conversioncharacteristic, and outputting the output signal, wherein the conversioncharacteristic is represented by a first tone curve, the first tonecurve has an input variable corresponding to the image signal and anoutput variable corresponding to the output signal, the first tone curvechanges according to the representative value, the first tone curve hasa property in which a rate of change of the output variable with respectto the input variable is greater than that of a second tone curve, inthe vicinity of a point of intersection between the first tone curve andthe second tone curve, and the second tone curve represents arelationship between the input variable and the output variableoutputted when the image signal is equal to the representative value.